Oct 29 22:39:56.318924 ip-10-0-143-172 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 29 22:39:56.318955 ip-10-0-143-172 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 29 22:39:56.318964 ip-10-0-143-172 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 29 22:39:56.319280 ip-10-0-143-172 systemd[1]: Failed to start Kubernetes Kubelet. Oct 29 22:40:06.494030 ip-10-0-143-172 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 29 22:40:06.494047 ip-10-0-143-172 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6c5608ce3c4b4bec8b0413edfa6e5e2b -- Oct 29 22:42:09.868452 ip-10-0-143-172 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:42:10.246790 ip-10-0-143-172 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:42:10.246790 ip-10-0-143-172 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:42:10.246790 ip-10-0-143-172 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:42:10.246790 ip-10-0-143-172 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:42:10.251724 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251642 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:10.251724 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251721 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:10.251724 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251725 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:10.251724 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251729 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:10.251724 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251732 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251744 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251748 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251753 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251757 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251761 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251764 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251767 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251770 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251773 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251776 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251779 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251781 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251784 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251787 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251790 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251792 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251795 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251797 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:10.251929 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251800 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251802 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251805 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251807 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251810 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251813 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251815 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251818 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251822 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251825 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251828 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251831 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251833 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251836 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251839 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251842 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251844 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251847 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251849 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251852 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:10.252386 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251855 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251857 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251860 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251862 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251866 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251870 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251873 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251877 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251881 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251884 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251887 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251889 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251892 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251894 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251897 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251900 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251902 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251906 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251908 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251911 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:10.252858 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251914 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251916 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251919 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251922 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251925 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251928 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251931 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251934 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251937 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251940 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251943 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251945 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251948 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251951 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251953 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251956 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251958 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251961 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251963 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:10.253371 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251966 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251970 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251972 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251975 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251977 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251980 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251983 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251985 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251987 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251990 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251993 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251996 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.251998 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252001 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252005 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252008 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252010 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252013 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252016 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252018 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:10.253828 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252021 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:10.254318 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252024 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:10.254318 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252026 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:10.254318 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.252029 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:10.254504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254491 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:10.254504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254503 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254508 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254511 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254515 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254518 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254522 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254524 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254528 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254530 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254534 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254537 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254540 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254542 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254545 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254548 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254550 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254553 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254555 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254558 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254561 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:10.254555 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254564 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254567 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254570 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254572 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254575 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254577 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254580 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254583 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254585 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254587 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254590 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254593 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254595 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254598 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254600 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254604 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254606 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254609 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254611 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:10.255063 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254614 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254617 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254619 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254623 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254626 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254628 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254631 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254633 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254636 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254638 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254641 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254643 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254646 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254648 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254651 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254655 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254658 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254660 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254663 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254665 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:10.255549 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254668 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254670 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254673 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254676 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254678 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254681 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254683 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254686 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254688 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254691 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254693 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254696 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254699 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254701 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254705 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254709 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254712 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254715 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254718 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:10.256078 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254721 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254724 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254726 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254730 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254734 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254737 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254740 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254743 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254746 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254749 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254752 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254755 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254757 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254760 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254762 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254765 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254767 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254770 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254773 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:10.256558 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254775 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254778 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254780 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254783 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254785 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254788 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254791 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.254793 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255303 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255309 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255313 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255316 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255319 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255321 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255324 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255327 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255329 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255332 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255335 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255337 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:10.257088 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255340 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255343 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255346 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255348 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255351 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255354 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255357 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255360 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255362 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255365 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255367 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255370 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255373 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255375 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255378 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255380 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255383 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255385 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255388 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:10.257576 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255391 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255393 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255395 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255398 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255401 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255403 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255407 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255411 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255414 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255416 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255420 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255422 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255425 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255427 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255431 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255434 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255436 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255439 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255441 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:10.258018 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255444 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255447 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255449 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255452 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255454 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255457 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255459 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255462 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255465 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255467 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255470 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255472 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255475 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255477 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255480 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255482 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255485 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255487 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255490 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255493 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:10.258479 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255495 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255498 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255500 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255504 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255508 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255511 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255514 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255518 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255521 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255523 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255526 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255529 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255531 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255534 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255536 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255539 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255542 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255544 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255547 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255549 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:10.258957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255552 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255555 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255558 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255560 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255563 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255566 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255568 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255571 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255573 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255576 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255578 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255582 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255585 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255588 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255590 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.255593 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256521 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256531 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256537 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256544 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:42:10.259447 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256548 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256552 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256556 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256561 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256564 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256567 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256571 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256575 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256578 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256581 2575 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256584 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256587 2575 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256590 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256593 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256598 2575 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256600 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256604 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256607 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256610 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256614 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256617 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256620 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256623 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256626 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:42:10.259919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256630 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256634 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256637 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256640 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256644 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256647 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256650 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256654 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256657 2575 flags.go:64] FLAG: --enable-server="true" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256660 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256664 2575 flags.go:64] FLAG: --event-burst="100" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256668 2575 flags.go:64] FLAG: --event-qps="50" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256671 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256674 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256677 2575 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256681 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256684 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256687 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256690 2575 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256693 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256696 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256699 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256702 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256705 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256708 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:42:10.260528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256711 2575 flags.go:64] FLAG: --feature-gates="" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256715 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256718 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256721 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256724 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256727 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256731 2575 flags.go:64] FLAG: --help="false" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256734 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256738 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256741 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256744 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256747 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256751 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256754 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256757 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256760 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256763 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256766 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256769 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256772 2575 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256775 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256778 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256781 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256784 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:42:10.261133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256787 2575 flags.go:64] FLAG: --lock-file="" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256790 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256793 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256796 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256801 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256804 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256807 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256810 2575 flags.go:64] FLAG: --logging-format="text" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256814 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256817 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256820 2575 flags.go:64] FLAG: --manifest-url="" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256823 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256827 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256830 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256834 2575 flags.go:64] FLAG: --max-pods="110" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256837 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256841 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256845 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256848 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256851 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256853 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256856 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256863 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256867 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256870 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:42:10.261737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256873 2575 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256876 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256879 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256882 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256885 2575 flags.go:64] FLAG: --port="10250" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256888 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256891 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-093c7071a13097967" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256894 2575 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256897 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256900 2575 flags.go:64] FLAG: --register-node="true" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256903 2575 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256907 2575 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256909 2575 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256912 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256915 2575 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256919 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256922 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256925 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256928 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256931 2575 flags.go:64] FLAG: --runonce="false" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256934 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256937 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256940 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256943 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256946 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256949 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:42:10.262351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256952 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256956 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256958 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256962 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256965 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256968 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256971 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256974 2575 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256977 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256982 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256985 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256987 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256992 2575 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256995 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.256997 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.257000 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.257003 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.257006 2575 flags.go:64] FLAG: --v="2" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.257011 2575 flags.go:64] FLAG: --version="false" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.257015 2575 flags.go:64] FLAG: --vmodule="" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.257018 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.257022 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257150 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257154 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:10.263101 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257157 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257160 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257165 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257168 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257171 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257173 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257176 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257179 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257182 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257185 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257188 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257192 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257195 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257198 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257214 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257217 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257220 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257223 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257226 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257228 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:10.263692 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257231 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257234 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257236 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257239 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257242 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257245 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257247 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257250 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257252 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257255 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257258 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257260 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257263 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257266 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257270 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257272 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257275 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257277 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257280 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257283 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:10.264176 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257285 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257288 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257291 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257295 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257299 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257304 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257307 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257311 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257313 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257317 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257319 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257322 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257325 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257328 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257330 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257333 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257335 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257338 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257341 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:10.264659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257343 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257346 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257348 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257351 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257353 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257356 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257359 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257362 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257365 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257367 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257370 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257373 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257375 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257378 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257380 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257384 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257388 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257391 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257393 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257396 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:10.265144 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257398 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257401 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257405 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257409 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257412 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257414 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257417 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257419 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257422 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257425 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257427 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257430 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257433 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257435 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257438 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257441 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257443 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257446 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257448 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:10.265635 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257452 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257455 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257458 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257460 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257463 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257465 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.257516 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257551 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257555 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257560 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257563 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257567 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257570 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257573 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257576 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:10.266086 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257579 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257582 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257585 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257588 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257590 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257593 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257596 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257599 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257601 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257604 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257607 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257609 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257612 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257615 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257617 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257620 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257622 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257625 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257629 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257631 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:10.266504 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257634 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257637 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257640 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257642 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257645 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257647 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257651 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257654 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257657 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257660 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257662 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257665 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257668 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257671 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257673 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257676 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257679 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257682 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257685 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257687 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:10.266987 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257690 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257692 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257695 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257697 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257700 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257703 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257705 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257708 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257711 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257713 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257717 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257719 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257722 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257725 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257727 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257730 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257733 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257735 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257739 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257742 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:10.267468 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257745 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257747 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257750 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257753 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257755 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257760 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257763 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257766 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257769 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257772 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257775 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257777 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257780 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257783 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257785 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257788 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257791 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257794 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257796 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:10.267943 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257799 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257801 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257804 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257808 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257810 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257813 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257816 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257818 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257821 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257824 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257826 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257830 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257832 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257835 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257838 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257841 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257844 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257847 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.257849 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.264281 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:42:10.268419 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.264296 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.264311 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.264316 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264380 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264386 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264390 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264393 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264396 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264400 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264405 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264408 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264411 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264414 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264417 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264419 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264422 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264425 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264427 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264430 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:10.268920 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264433 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264435 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264438 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264440 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264443 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264445 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264449 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264458 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264461 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264463 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264466 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264468 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264471 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264474 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264477 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264480 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264483 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264485 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264488 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264491 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:10.269428 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264493 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264496 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264499 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264501 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264504 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264506 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264509 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264511 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264514 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264517 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264519 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264522 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264525 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264527 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264530 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264532 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264535 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264538 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264541 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264543 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:10.269999 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264552 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264554 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264557 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264560 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264562 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264580 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264584 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264587 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264590 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264593 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264596 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264599 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264601 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264604 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264607 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264609 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264612 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264614 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264617 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:10.270554 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264620 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264623 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264625 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264628 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264631 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264634 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264636 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264639 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264642 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264644 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264647 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264650 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264652 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264655 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264664 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264669 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264672 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264675 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264678 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264680 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:10.271035 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264683 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264686 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264688 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264691 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264694 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264696 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264699 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264701 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264704 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264706 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264709 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.264737 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264927 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264932 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264935 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264937 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:10.271899 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264940 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264942 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264945 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264947 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264950 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264952 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264955 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264958 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264960 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264963 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264966 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264974 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264977 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264980 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264982 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264985 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264988 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264990 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264993 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264995 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:10.272513 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.264998 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265000 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265003 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265006 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265008 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265011 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265013 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265016 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265019 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265021 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265024 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265027 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265029 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265033 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265037 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265040 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265043 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265045 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265048 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:10.273261 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265050 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265053 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265057 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265061 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265064 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265073 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265075 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265078 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265081 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265084 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265087 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265089 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265092 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265094 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265097 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265099 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265102 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265104 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265107 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:10.273762 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265109 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265112 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265114 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265117 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265119 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265122 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265124 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265127 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265129 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265132 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265134 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265137 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265140 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265142 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265145 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265148 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265151 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265153 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265156 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265164 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:10.274232 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265167 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265170 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265173 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265175 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265178 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265180 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265183 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265185 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265193 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265196 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265214 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265217 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265220 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265222 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265225 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265227 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265230 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265233 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265235 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:10.274686 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265238 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265240 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265243 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265245 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265248 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.265272 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265322 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265326 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265329 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265332 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265335 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265337 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265340 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265348 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265351 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265354 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:10.275352 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265357 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265359 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265362 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265364 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265367 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265369 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265372 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265375 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265377 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265380 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265382 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265388 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265391 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265394 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265396 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265399 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265401 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265404 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265406 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265409 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:10.275957 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265412 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265417 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265419 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265422 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265425 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265427 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265430 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265432 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265435 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265438 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265446 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265449 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265452 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265455 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265459 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265463 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265466 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265468 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265471 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:10.276533 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265473 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265476 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265478 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265482 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265486 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265488 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265491 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265494 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265496 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265499 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265502 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265504 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265506 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265509 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265512 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265517 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265520 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265523 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265525 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265528 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:10.277034 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265530 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265533 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265535 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265538 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265546 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265549 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265552 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265554 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265557 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265559 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265562 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265565 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265567 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265570 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265572 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265575 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265577 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265580 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265582 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265585 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:10.277637 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265587 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265590 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265593 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265595 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265598 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265600 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265604 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265608 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265610 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265613 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265615 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265618 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265621 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265628 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265631 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265633 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:10.265636 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.265688 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.268372 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 22:42:10.278121 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.269318 2575 server.go:1014] "Starting client certificate rotation" Oct 29 22:42:10.278587 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.269434 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:42:10.278587 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.269478 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:42:10.278587 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.271566 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:42:10.278587 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.276801 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:42:10.288034 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.288020 2575 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:42:10.291173 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.291158 2575 log.go:25] "Validated CRI v1 image API" Oct 29 22:42:10.291963 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.291935 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:42:10.298246 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.298226 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d877c562-a4a7-4bf9-8837-e2523146b187:/dev/nvme0n1p3 dc5250ab-f587-4dfa-b14a-60b9f1e90bc6:/dev/nvme0n1p4] Oct 29 22:42:10.298340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.298244 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 22:42:10.303809 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.303694 2575 manager.go:217] Machine: {Timestamp:2026-10-29 22:42:10.302466443 +0000 UTC m=+0.324710869 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3229396 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25bde06274256c634eeeb1d4acc638 SystemUUID:ec25bde0-6274-256c-634e-eeb1d4acc638 BootID:6c5608ce-3c4b-4bec-8b04-13edfa6e5e2b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:df:93:1a:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:df:93:1a:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:98:90:80:e7:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:42:10.303877 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.303810 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:42:10.303912 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.303878 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:42:10.306133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.306114 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:42:10.306263 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.306134 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:42:10.306305 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.306266 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:42:10.306305 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.306275 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:42:10.306397 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.306310 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:42:10.306397 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.306323 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:42:10.307076 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.307066 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:42:10.310258 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.310246 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:42:10.310367 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.310359 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:42:10.311238 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.311230 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:42:10.311276 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.311242 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:42:10.311276 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.311251 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:42:10.311276 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.311262 2575 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:42:10.311276 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.311274 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:42:10.311572 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.311560 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:42:10.313462 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.313441 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:42:10.319032 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.319016 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:42:10.319628 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.319613 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:42:10.319687 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.319644 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:42:10.319817 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.319804 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:42:10.319849 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.319823 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:42:10.319849 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.319829 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:42:10.319849 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.319835 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:42:10.319849 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.319840 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:42:10.319849 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.319847 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:42:10.320345 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.320336 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:42:10.320388 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.320346 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:42:10.320388 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.320353 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:42:10.320388 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.320359 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:42:10.320388 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.320368 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:42:10.320388 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.320377 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:42:10.321513 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.321503 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:42:10.321563 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.321515 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:42:10.326250 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.326236 2575 server.go:1263] "Started kubelet" Oct 29 22:42:10.326356 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.326297 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:42:10.326407 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.326314 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:42:10.326407 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.326390 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:42:10.326896 ip-10-0-143-172 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:42:10.327639 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.327621 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:42:10.328471 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.328344 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:42:10.332806 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.332783 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 22:42:10.333225 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.333212 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:42:10.333648 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.333632 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:42:10.334126 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.333771 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:10.334306 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334274 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:42:10.334306 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334275 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:42:10.334434 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334318 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:42:10.334434 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334416 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:42:10.334434 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334425 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:42:10.334513 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334447 2575 factory.go:55] Registering systemd factory Oct 29 22:42:10.334513 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334459 2575 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:42:10.334662 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334651 2575 factory.go:153] Registering CRI-O factory Oct 29 22:42:10.334711 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334664 2575 factory.go:223] Registration of the crio container factory successfully Oct 29 22:42:10.334711 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334699 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:42:10.334812 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334719 2575 factory.go:103] Registering Raw factory Oct 29 22:42:10.334812 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.334731 2575 manager.go:1199] Started watching for new ooms in manager Oct 29 22:42:10.335789 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.335773 2575 manager.go:319] Starting recovery of all containers Oct 29 22:42:10.347732 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.347697 2575 manager.go:324] Recovery completed Oct 29 22:42:10.354176 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.354158 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:42:10.355744 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.355723 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:42:10.355810 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.355760 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:42:10.355810 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.355771 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:42:10.359226 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.356247 2575 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:42:10.359226 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.356259 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:42:10.359226 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.356270 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:42:10.359226 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.356277 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:42:10.359226 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.356966 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:42:10.359226 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.356991 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:42:10.359226 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.357032 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:42:10.366287 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.366259 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 22:42:10.366287 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.366257 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 22:42:10.367557 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.366251 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-172.ec2.internal.18e321561aec47f3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-172.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-172.ec2.internal,},FirstTimestamp:2026-10-29 22:42:10.326185971 +0000 UTC m=+0.348430393,LastTimestamp:2026-10-29 22:42:10.326185971 +0000 UTC m=+0.348430393,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-172.ec2.internal,}" Oct 29 22:42:10.367557 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.367550 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:42:10.376908 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.376882 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-59dm5" Oct 29 22:42:10.382065 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.382045 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 22:42:10.382065 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.382054 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 22:42:10.382162 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.382045 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 22:42:10.387169 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.387153 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-59dm5" Oct 29 22:42:10.409484 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.409409 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-172.ec2.internal.18e321561caf4e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-172.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-172.ec2.internal,},FirstTimestamp:2026-10-29 22:42:10.355744384 +0000 UTC m=+0.377988813,LastTimestamp:2026-10-29 22:42:10.355744384 +0000 UTC m=+0.377988813,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-172.ec2.internal,}" Oct 29 22:42:10.433245 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.433164 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-172.ec2.internal.18e321561cafa114 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-172.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-172.ec2.internal,},FirstTimestamp:2026-10-29 22:42:10.355765524 +0000 UTC m=+0.378009946,LastTimestamp:2026-10-29 22:42:10.355765524 +0000 UTC m=+0.378009946,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-172.ec2.internal,}" Oct 29 22:42:10.434702 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.434684 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:10.459282 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.459219 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-172.ec2.internal.18e321561cafc3c3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-172.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-172.ec2.internal,},FirstTimestamp:2026-10-29 22:42:10.355774403 +0000 UTC m=+0.378018826,LastTimestamp:2026-10-29 22:42:10.355774403 +0000 UTC m=+0.378018826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-172.ec2.internal,}" Oct 29 22:42:10.481828 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.481759 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-172.ec2.internal.18e321561caf4e80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-172.ec2.internal.18e321561caf4e80 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-172.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-172.ec2.internal,},FirstTimestamp:2026-10-29 22:42:10.355744384 +0000 UTC m=+0.377988813,LastTimestamp:2026-10-29 22:42:10.356977989 +0000 UTC m=+0.379222411,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-172.ec2.internal,}" Oct 29 22:42:10.496863 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.496812 2575 policy_none.go:50] "Start" Oct 29 22:42:10.496863 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.496834 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:42:10.496863 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.496849 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:42:10.497912 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.497902 2575 policy_none.go:44] "Start" Oct 29 22:42:10.517146 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.516943 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-172.ec2.internal.18e321561cafa114\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-172.ec2.internal.18e321561cafa114 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-172.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-172.ec2.internal,},FirstTimestamp:2026-10-29 22:42:10.355765524 +0000 UTC m=+0.378009946,LastTimestamp:2026-10-29 22:42:10.356995459 +0000 UTC m=+0.379239881,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-172.ec2.internal,}" Oct 29 22:42:10.534835 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.534819 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.536682 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.539004 2575 manager.go:341] "Starting Device Plugin manager" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.539031 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.539041 2575 server.go:87] "Starting device plugin registration server" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.539243 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.539253 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.539338 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.539503 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.539530 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.540012 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:42:10.543010 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.540054 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:10.574851 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.574822 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:42:10.575954 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.575939 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:42:10.576022 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.575960 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:42:10.576022 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.575975 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:42:10.576022 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.576007 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 22:42:10.590646 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.590624 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:10.595802 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.595785 2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-172.ec2.internal\" not found" node="ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.639498 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.639481 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:42:10.640305 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.640278 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:42:10.640380 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.640319 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:42:10.640380 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.640333 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:42:10.640380 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.640356 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.659269 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.659251 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.659321 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.659271 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-172.ec2.internal\": node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:10.676604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.676572 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal"] Oct 29 22:42:10.676653 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.676644 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:42:10.677456 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.677439 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:42:10.677567 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.677472 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:42:10.677567 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.677485 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:42:10.678737 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.678720 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:42:10.678946 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.678931 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.679004 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.678992 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:42:10.679511 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.679487 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:42:10.679586 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.679520 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:42:10.679586 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.679535 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:42:10.679655 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.679591 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:42:10.679655 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.679613 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:42:10.679655 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.679625 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:42:10.680841 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.680827 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.680880 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.680854 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:42:10.681623 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.681607 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:42:10.681686 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.681632 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:42:10.681686 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.681662 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:42:10.704369 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.704350 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:10.715711 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.715693 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-172.ec2.internal\" not found" node="ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.720459 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.720381 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-172.ec2.internal\" not found" node="ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.735835 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.735808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.735835 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.735831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.735961 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.735850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3d32c15ddf0c3d5f6df2743c0d67b79-config\") pod \"kube-apiserver-proxy-ip-10-0-143-172.ec2.internal\" (UID: \"f3d32c15ddf0c3d5f6df2743c0d67b79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.805241 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.805185 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:10.836287 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.836269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3d32c15ddf0c3d5f6df2743c0d67b79-config\") pod \"kube-apiserver-proxy-ip-10-0-143-172.ec2.internal\" (UID: \"f3d32c15ddf0c3d5f6df2743c0d67b79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.836361 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.836291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.836361 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.836307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.836361 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.836344 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.836460 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.836382 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3d32c15ddf0c3d5f6df2743c0d67b79-config\") pod \"kube-apiserver-proxy-ip-10-0-143-172.ec2.internal\" (UID: \"f3d32c15ddf0c3d5f6df2743c0d67b79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.836460 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:10.836382 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:42:10.905724 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:10.905704 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:11.006768 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:11.006737 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:11.018905 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:11.018888 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:42:11.023805 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:11.023790 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:42:11.107224 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:11.107154 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:11.207635 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:11.207606 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:11.269036 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:11.269016 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 22:42:11.269448 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:11.269125 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:42:12.369713 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:12.369693 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:12.470006 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:12.469985 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:12.561281 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.561243 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:37:11 +0000 UTC" deadline="2028-04-02 23:14:07.322490318 +0000 UTC" Oct 29 22:42:12.561440 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.561287 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12504h31m54.761217874s" Oct 29 22:42:12.570533 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:12.570510 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:42:12.582706 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.582689 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:12.614129 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.614113 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:12.623557 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:12.623530 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d509ddddb3f91e54ef73594cf479892.slice/crio-f7a8fd2af6af21a4139d5b7c6d92fd2224131579fcc42e1784ae607e72b93edc WatchSource:0}: Error finding container f7a8fd2af6af21a4139d5b7c6d92fd2224131579fcc42e1784ae607e72b93edc: Status 404 returned error can't find the container with id f7a8fd2af6af21a4139d5b7c6d92fd2224131579fcc42e1784ae607e72b93edc Oct 29 22:42:12.623982 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:12.623963 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d32c15ddf0c3d5f6df2743c0d67b79.slice/crio-5847add22193d273cbe18ef95c4be2741a72265fe0db69c237b7a86b5ac15b51 WatchSource:0}: Error finding container 5847add22193d273cbe18ef95c4be2741a72265fe0db69c237b7a86b5ac15b51: Status 404 returned error can't find the container with id 5847add22193d273cbe18ef95c4be2741a72265fe0db69c237b7a86b5ac15b51 Oct 29 22:42:12.628824 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.628808 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:42:12.640490 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.640451 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" event={"ID":"f3d32c15ddf0c3d5f6df2743c0d67b79","Type":"ContainerStarted","Data":"5847add22193d273cbe18ef95c4be2741a72265fe0db69c237b7a86b5ac15b51"} Oct 29 22:42:12.641494 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.641473 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" event={"ID":"3d509ddddb3f91e54ef73594cf479892","Type":"ContainerStarted","Data":"f7a8fd2af6af21a4139d5b7c6d92fd2224131579fcc42e1784ae607e72b93edc"} Oct 29 22:42:12.696049 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.696028 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:42:12.721644 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.721603 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:12.723300 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.723288 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:42:12.746303 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.746286 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:12.931974 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:12.931949 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:13.373372 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.373117 2575 apiserver.go:52] "Watching apiserver" Oct 29 22:42:13.395952 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.395919 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 22:42:13.420893 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.420868 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:42:13.421415 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.421390 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-ncjz6","openshift-multus/multus-cpp5c","kube-system/konnectivity-agent-lgqrb","openshift-multus/multus-additional-cni-plugins-ptfk2","openshift-multus/network-metrics-daemon-d286w","openshift-ovn-kubernetes/ovnkube-node-vppzr","kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal","openshift-network-diagnostics/network-check-target-lglpm","openshift-cluster-node-tuning-operator/tuned-jk6th","openshift-network-operator/iptables-alerter-dcjkg","kube-system/global-pull-secret-syncer-twb2h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp","openshift-dns/node-resolver-5rgg6"] Oct 29 22:42:13.423928 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.423905 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.426038 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.426016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.428235 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.428212 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:13.430539 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.430510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.433170 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.432802 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:13.433170 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.432886 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:13.435153 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.435132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.437333 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.437315 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:13.437424 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.437370 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:13.440336 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.440310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.441476 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.441452 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:13.441592 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.441476 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xjmd7\"" Oct 29 22:42:13.442146 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.442126 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:13.442423 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.442403 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:42:13.442529 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.442513 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.442760 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.442731 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-phcjr\"" Oct 29 22:42:13.443135 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.443118 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:42:13.443198 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.443145 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:42:13.444999 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.444979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.445132 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.445107 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:42:13.445417 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.445402 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:42:13.445507 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.445456 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fchfg\"" Oct 29 22:42:13.447224 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.447207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.449652 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.449632 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:13.449743 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.449699 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:13.454830 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.454806 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:42:13.454923 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.454867 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-29z7s" Oct 29 22:42:13.459727 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.459706 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-29z7s" Oct 29 22:42:13.460217 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.460200 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:13.497105 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.497090 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:42:13.505216 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.505330 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11cc5-35ab-4347-8914-e953b71bc322-konnectivity-ca\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:13.505330 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-conf-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.505330 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr776\" (UniqueName: \"kubernetes.io/projected/6a6a57a5-af08-4ceb-a886-c176359b8d5a-kube-api-access-lr776\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.505494 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cnibin\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.505494 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx6bq\" (UniqueName: \"kubernetes.io/projected/b78e901b-2f4b-4ee5-a96d-573988c5cadd-kube-api-access-rx6bq\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.505610 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-ovn\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.505610 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-config\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.505610 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-kubelet-config\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:13.505743 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-tmp\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.505743 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-bin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.505743 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-var-lib-kubelet\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.505888 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505739 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-registration-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.505888 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b726c8b-651c-4d5e-afa5-26a9f159d609-hosts-file\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.505888 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpt7k\" (UniqueName: \"kubernetes.io/projected/8b726c8b-651c-4d5e-afa5-26a9f159d609-kube-api-access-vpt7k\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.505888 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.505888 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-systemd-units\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.505888 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.506120 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-run\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.506120 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-socket-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.506120 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfgr\" (UniqueName: \"kubernetes.io/projected/e85effca-b850-4af4-9086-ccfd965b7934-kube-api-access-jwfgr\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:13.506120 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.505981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-system-cni-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.506120 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-systemd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.506120 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-log-socket\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.506120 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506080 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.506120 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2sm9\" (UniqueName: \"kubernetes.io/projected/878dc633-6f09-450f-ab79-8788f14e3055-kube-api-access-s2sm9\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.506340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506123 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54a953c6-9194-404e-b8af-7ef3540d0e6d-iptables-alerter-script\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.506340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506174 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:13.506340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-socket-dir-parent\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.506340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.506340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-systemd\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.506340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-lib-modules\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.506340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-tuned\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.506340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b726c8b-651c-4d5e-afa5-26a9f159d609-tmp-dir\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.506340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-multus-certs\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-slash\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-modprobe-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-device-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506446 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fd11cc5-35ab-4347-8914-e953b71bc322-agent-certs\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-system-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-netns\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-dbus\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:13.506601 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-host\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506701 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-env-overrides\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtq46\" (UniqueName: \"kubernetes.io/projected/d839fc45-f2f4-4ce4-95d8-9e3145342973-kube-api-access-rtq46\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysconfig\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-hostroot\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cni-binary-copy\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-bin\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-script-lib\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kkhw\" (UniqueName: \"kubernetes.io/projected/a66d8310-cc78-4a48-bc21-03fa742d0829-kube-api-access-6kkhw\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-conf\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-netns\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-etc-kubernetes\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.507042 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cnibin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/878dc633-6f09-450f-ab79-8788f14e3055-host\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.506989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/878dc633-6f09-450f-ab79-8788f14e3055-serviceca\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x79c\" (UniqueName: \"kubernetes.io/projected/54a953c6-9194-404e-b8af-7ef3540d0e6d-kube-api-access-5x79c\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507027 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-multus\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-kubelet\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-var-lib-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-netd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-kubernetes\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-sys-fs\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-kubelet\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507288 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-daemon-config\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507319 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-node-log\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovn-node-metrics-cert\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-sys\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.507604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-k8s-cni-cncf-io\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.508181 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-etc-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.508181 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54a953c6-9194-404e-b8af-7ef3540d0e6d-host-slash\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.508181 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.507494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95k69\" (UniqueName: \"kubernetes.io/projected/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-kube-api-access-95k69\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.542025 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.541840 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:42:13.542025 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.541938 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:42:13.542025 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.541945 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:42:13.542288 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.542084 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:42:13.542288 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.542121 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:42:13.542378 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.542288 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:42:13.542425 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.542385 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:42:13.542425 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.542400 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:42:13.542494 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.542439 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:42:13.544009 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.543991 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:42:13.544137 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.544010 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rpfjl\"" Oct 29 22:42:13.544137 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.544084 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:42:13.545637 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.544653 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:42:13.545637 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.544763 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:42:13.545637 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.544896 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:13.545637 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.545133 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7jjw\"" Oct 29 22:42:13.545637 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.545263 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m52c5\"" Oct 29 22:42:13.545637 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.545451 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:13.545637 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.545530 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tj892\"" Oct 29 22:42:13.545995 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.545640 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:42:13.556429 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.556408 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:42:13.556537 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.556441 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:42:13.556537 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.556467 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2plp\"" Oct 29 22:42:13.556685 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.556605 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-28brx\"" Oct 29 22:42:13.556751 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.556685 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:42:13.558801 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.558783 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:42:13.608146 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cnibin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.608322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/878dc633-6f09-450f-ab79-8788f14e3055-host\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.608322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/878dc633-6f09-450f-ab79-8788f14e3055-serviceca\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.608322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x79c\" (UniqueName: \"kubernetes.io/projected/54a953c6-9194-404e-b8af-7ef3540d0e6d-kube-api-access-5x79c\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.608322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-multus\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.608322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608249 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/878dc633-6f09-450f-ab79-8788f14e3055-host\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.608322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-kubelet\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cnibin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.608322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-kubelet\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608323 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-multus\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-var-lib-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-netd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-kubernetes\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-sys-fs\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-kubernetes\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608622 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-netd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608642 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-var-lib-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-kubelet\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.608692 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-daemon-config\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608730 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/878dc633-6f09-450f-ab79-8788f14e3055-serviceca\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-node-log\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovn-node-metrics-cert\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608814 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-sys-fs\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-sys\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608870 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-node-log\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-k8s-cni-cncf-io\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-etc-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608924 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-kubelet\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54a953c6-9194-404e-b8af-7ef3540d0e6d-host-slash\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608956 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-k8s-cni-cncf-io\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.608975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.608959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95k69\" (UniqueName: \"kubernetes.io/projected/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-kube-api-access-95k69\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.609297 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609004 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-etc-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.609297 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.609297 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11cc5-35ab-4347-8914-e953b71bc322-konnectivity-ca\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:13.609297 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-conf-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.609297 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609068 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:42:13.609297 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609099 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.609297 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609147 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-sys\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.609297 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609264 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54a953c6-9194-404e-b8af-7ef3540d0e6d-host-slash\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.609565 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609347 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-conf-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.609565 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr776\" (UniqueName: \"kubernetes.io/projected/6a6a57a5-af08-4ceb-a886-c176359b8d5a-kube-api-access-lr776\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.609565 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cnibin\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.609754 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx6bq\" (UniqueName: \"kubernetes.io/projected/b78e901b-2f4b-4ee5-a96d-573988c5cadd-kube-api-access-rx6bq\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.609754 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-ovn\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.609754 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-config\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.609754 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-kubelet-config\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:13.609754 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-tmp\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.609754 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-bin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.609754 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-var-lib-kubelet\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.609754 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-registration-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b726c8b-651c-4d5e-afa5-26a9f159d609-hosts-file\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpt7k\" (UniqueName: \"kubernetes.io/projected/8b726c8b-651c-4d5e-afa5-26a9f159d609-kube-api-access-vpt7k\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609805 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-daemon-config\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-systemd-units\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609847 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11cc5-35ab-4347-8914-e953b71bc322-konnectivity-ca\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-run\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-socket-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609910 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-bin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfgr\" (UniqueName: \"kubernetes.io/projected/e85effca-b850-4af4-9086-ccfd965b7934-kube-api-access-jwfgr\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-system-cni-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-systemd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.609996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-log-socket\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2sm9\" (UniqueName: \"kubernetes.io/projected/878dc633-6f09-450f-ab79-8788f14e3055-kube-api-access-s2sm9\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.610125 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54a953c6-9194-404e-b8af-7ef3540d0e6d-iptables-alerter-script\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610084 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-var-lib-kubelet\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610136 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-socket-dir-parent\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.610167 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-systemd\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.610232 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret podName:fb913afd-66ca-46e3-b9d1-66cff498245d nodeName:}" failed. No retries permitted until 2026-10-29 22:42:14.110213473 +0000 UTC m=+3.071080065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret") pod "global-pull-secret-syncer-twb2h" (UID: "fb913afd-66ca-46e3-b9d1-66cff498245d") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-lib-modules\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-systemd\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-tuned\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610306 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-registration-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610318 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-system-cni-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610323 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cnibin\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b726c8b-651c-4d5e-afa5-26a9f159d609-tmp-dir\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610352 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-ovn\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-multus-certs\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.610891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-slash\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-modprobe-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-device-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fd11cc5-35ab-4347-8914-e953b71bc322-agent-certs\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-system-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610617 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-systemd-units\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-netns\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-netns\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-dbus\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-host\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-systemd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610741 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-log-socket\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.611690 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610783 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-env-overrides\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtq46\" (UniqueName: \"kubernetes.io/projected/d839fc45-f2f4-4ce4-95d8-9e3145342973-kube-api-access-rtq46\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysconfig\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610876 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-hostroot\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cni-binary-copy\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-bin\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-script-lib\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610959 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.610976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kkhw\" (UniqueName: \"kubernetes.io/projected/a66d8310-cc78-4a48-bc21-03fa742d0829-kube-api-access-6kkhw\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-conf\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-netns\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611059 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-socket-dir-parent\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-etc-kubernetes\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611154 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-etc-kubernetes\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.612445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611198 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611202 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-kubelet-config\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611250 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-run\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611294 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysconfig\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611304 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-dbus\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611334 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-hostroot\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-host\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.611421 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611422 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-config\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611461 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54a953c6-9194-404e-b8af-7ef3540d0e6d-iptables-alerter-script\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.611498 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs podName:e85effca-b850-4af4-9086-ccfd965b7934 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:14.111482701 +0000 UTC m=+3.072349279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs") pod "network-metrics-daemon-d286w" (UID: "e85effca-b850-4af4-9086-ccfd965b7934") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-lib-modules\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611878 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-env-overrides\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611910 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.611931 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cni-binary-copy\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612044 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612039 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-conf\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.613237 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612062 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612105 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-system-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612116 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b726c8b-651c-4d5e-afa5-26a9f159d609-hosts-file\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612225 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-bin\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612282 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-slash\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612330 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-multus-certs\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612350 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-socket-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612374 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612399 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b726c8b-651c-4d5e-afa5-26a9f159d609-tmp-dir\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612462 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-modprobe-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612472 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-netns\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612520 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-device-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.612847 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-script-lib\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.613892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.613321 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-tmp\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.614505 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.614458 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovn-node-metrics-cert\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.614873 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.614849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-tuned\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.615059 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.615043 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fd11cc5-35ab-4347-8914-e953b71bc322-agent-certs\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:13.634046 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.633989 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:13.634046 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.634015 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:13.634046 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.634027 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2skn4 for pod openshift-network-diagnostics/network-check-target-lglpm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:13.634243 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:13.634085 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4 podName:7d4c20d8-f5e8-450b-b4f4-8199b277bbb0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:14.134068198 +0000 UTC m=+3.094934790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2skn4" (UniqueName: "kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4") pod "network-check-target-lglpm" (UID: "7d4c20d8-f5e8-450b-b4f4-8199b277bbb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:13.637181 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.637154 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpt7k\" (UniqueName: \"kubernetes.io/projected/8b726c8b-651c-4d5e-afa5-26a9f159d609-kube-api-access-vpt7k\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.637274 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.637216 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2sm9\" (UniqueName: \"kubernetes.io/projected/878dc633-6f09-450f-ab79-8788f14e3055-kube-api-access-s2sm9\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.642167 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.642070 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95k69\" (UniqueName: \"kubernetes.io/projected/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-kube-api-access-95k69\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.642861 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.642843 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfgr\" (UniqueName: \"kubernetes.io/projected/e85effca-b850-4af4-9086-ccfd965b7934-kube-api-access-jwfgr\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:13.643438 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.643406 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kkhw\" (UniqueName: \"kubernetes.io/projected/a66d8310-cc78-4a48-bc21-03fa742d0829-kube-api-access-6kkhw\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.643835 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.643810 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x79c\" (UniqueName: \"kubernetes.io/projected/54a953c6-9194-404e-b8af-7ef3540d0e6d-kube-api-access-5x79c\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.645169 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.645148 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr776\" (UniqueName: \"kubernetes.io/projected/6a6a57a5-af08-4ceb-a886-c176359b8d5a-kube-api-access-lr776\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.645589 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.645554 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx6bq\" (UniqueName: \"kubernetes.io/projected/b78e901b-2f4b-4ee5-a96d-573988c5cadd-kube-api-access-rx6bq\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.647247 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.647213 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtq46\" (UniqueName: \"kubernetes.io/projected/d839fc45-f2f4-4ce4-95d8-9e3145342973-kube-api-access-rtq46\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.741315 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.741282 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:42:13.752950 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:13.752920 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0087fb0_f6d1_4f82_b78b_7f460f997ef0.slice/crio-71f53435bd31fec098855868cf480f11c40f6e3ba94ae7383de87959bea59d54 WatchSource:0}: Error finding container 71f53435bd31fec098855868cf480f11c40f6e3ba94ae7383de87959bea59d54: Status 404 returned error can't find the container with id 71f53435bd31fec098855868cf480f11c40f6e3ba94ae7383de87959bea59d54 Oct 29 22:42:13.753052 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.753031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:42:13.758679 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.758495 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:13.763972 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:13.763363 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66d8310_cc78_4a48_bc21_03fa742d0829.slice/crio-6df744fbb0e4f3f560f8df3d294b089e974d659b20d2f3ae7e3d812ed6ae4e74 WatchSource:0}: Error finding container 6df744fbb0e4f3f560f8df3d294b089e974d659b20d2f3ae7e3d812ed6ae4e74: Status 404 returned error can't find the container with id 6df744fbb0e4f3f560f8df3d294b089e974d659b20d2f3ae7e3d812ed6ae4e74 Oct 29 22:42:13.781844 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.781826 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:42:13.784014 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.783989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:42:13.792689 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.792654 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cpp5c" Oct 29 22:42:13.795539 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:13.795518 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78e901b_2f4b_4ee5_a96d_573988c5cadd.slice/crio-ac8ec5ee4af5ab87210981e33ad8a03de9364f5ddc8cdbb5041a56d6a9e75b27 WatchSource:0}: Error finding container ac8ec5ee4af5ab87210981e33ad8a03de9364f5ddc8cdbb5041a56d6a9e75b27: Status 404 returned error can't find the container with id ac8ec5ee4af5ab87210981e33ad8a03de9364f5ddc8cdbb5041a56d6a9e75b27 Oct 29 22:42:13.796355 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:13.796296 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b726c8b_651c_4d5e_afa5_26a9f159d609.slice/crio-f4c14097f29a0b32410b03debe8ffa4f099aa6ee1d7b888c4c32ac2ba978f410 WatchSource:0}: Error finding container f4c14097f29a0b32410b03debe8ffa4f099aa6ee1d7b888c4c32ac2ba978f410: Status 404 returned error can't find the container with id f4c14097f29a0b32410b03debe8ffa4f099aa6ee1d7b888c4c32ac2ba978f410 Oct 29 22:42:13.800113 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.800091 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:13.801788 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.801769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:42:13.803156 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:13.803129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:42:13.803636 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:13.803608 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6a57a5_af08_4ceb_a886_c176359b8d5a.slice/crio-85b7a4c9fffe7efecd43ab2ccecb1c9b8ee8b0432a9c97d0246b279322296d09 WatchSource:0}: Error finding container 85b7a4c9fffe7efecd43ab2ccecb1c9b8ee8b0432a9c97d0246b279322296d09: Status 404 returned error can't find the container with id 85b7a4c9fffe7efecd43ab2ccecb1c9b8ee8b0432a9c97d0246b279322296d09 Oct 29 22:42:13.821227 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:13.821201 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878dc633_6f09_450f_ab79_8788f14e3055.slice/crio-9ed974afc27d40236bc9889521d4ec285ca2b3450459909f787fa680be34fb41 WatchSource:0}: Error finding container 9ed974afc27d40236bc9889521d4ec285ca2b3450459909f787fa680be34fb41: Status 404 returned error can't find the container with id 9ed974afc27d40236bc9889521d4ec285ca2b3450459909f787fa680be34fb41 Oct 29 22:42:13.821659 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:13.821629 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd839fc45_f2f4_4ce4_95d8_9e3145342973.slice/crio-cdac24c91b8bfb9688187aaef68eba4f4e4c0bc3452bf60aa6687219885cf893 WatchSource:0}: Error finding container cdac24c91b8bfb9688187aaef68eba4f4e4c0bc3452bf60aa6687219885cf893: Status 404 returned error can't find the container with id cdac24c91b8bfb9688187aaef68eba4f4e4c0bc3452bf60aa6687219885cf893 Oct 29 22:42:13.822642 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:13.822623 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a953c6_9194_404e_b8af_7ef3540d0e6d.slice/crio-8bd8cb41078d1d26e13751de8b68473b6d736c7f6cc741d982f6bb0cdf7a9b5d WatchSource:0}: Error finding container 8bd8cb41078d1d26e13751de8b68473b6d736c7f6cc741d982f6bb0cdf7a9b5d: Status 404 returned error can't find the container with id 8bd8cb41078d1d26e13751de8b68473b6d736c7f6cc741d982f6bb0cdf7a9b5d Oct 29 22:42:14.114051 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.113892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:14.114167 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:14.114060 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:14.114167 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.114123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:14.114285 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:14.114172 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs podName:e85effca-b850-4af4-9086-ccfd965b7934 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:15.114154135 +0000 UTC m=+4.075020720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs") pod "network-metrics-daemon-d286w" (UID: "e85effca-b850-4af4-9086-ccfd965b7934") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:14.114343 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:14.114292 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:14.114394 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:14.114359 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret podName:fb913afd-66ca-46e3-b9d1-66cff498245d nodeName:}" failed. No retries permitted until 2026-10-29 22:42:15.114340497 +0000 UTC m=+4.075207088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret") pod "global-pull-secret-syncer-twb2h" (UID: "fb913afd-66ca-46e3-b9d1-66cff498245d") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:14.214426 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.214391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:14.214819 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:14.214798 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:14.216398 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:14.214825 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:14.216398 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:14.214837 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2skn4 for pod openshift-network-diagnostics/network-check-target-lglpm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:14.216398 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:14.214886 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4 podName:7d4c20d8-f5e8-450b-b4f4-8199b277bbb0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:15.214869784 +0000 UTC m=+4.175736382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2skn4" (UniqueName: "kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4") pod "network-check-target-lglpm" (UID: "7d4c20d8-f5e8-450b-b4f4-8199b277bbb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:14.463303 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.463244 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:37:13 +0000 UTC" deadline="2028-05-11 14:48:16.425506529 +0000 UTC" Oct 29 22:42:14.463303 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.463303 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13432h6m1.962206679s" Oct 29 22:42:14.638565 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.638066 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:14.638565 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:14.638203 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:14.648910 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.648843 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" event={"ID":"b78e901b-2f4b-4ee5-a96d-573988c5cadd","Type":"ContainerStarted","Data":"ac8ec5ee4af5ab87210981e33ad8a03de9364f5ddc8cdbb5041a56d6a9e75b27"} Oct 29 22:42:14.654712 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.654626 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" event={"ID":"a66d8310-cc78-4a48-bc21-03fa742d0829","Type":"ContainerStarted","Data":"6df744fbb0e4f3f560f8df3d294b089e974d659b20d2f3ae7e3d812ed6ae4e74"} Oct 29 22:42:14.662417 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.662350 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jk6th" event={"ID":"f0087fb0-f6d1-4f82-b78b-7f460f997ef0","Type":"ContainerStarted","Data":"71f53435bd31fec098855868cf480f11c40f6e3ba94ae7383de87959bea59d54"} Oct 29 22:42:14.674191 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.674123 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dcjkg" event={"ID":"54a953c6-9194-404e-b8af-7ef3540d0e6d","Type":"ContainerStarted","Data":"8bd8cb41078d1d26e13751de8b68473b6d736c7f6cc741d982f6bb0cdf7a9b5d"} Oct 29 22:42:14.687118 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.687061 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ncjz6" event={"ID":"878dc633-6f09-450f-ab79-8788f14e3055","Type":"ContainerStarted","Data":"9ed974afc27d40236bc9889521d4ec285ca2b3450459909f787fa680be34fb41"} Oct 29 22:42:14.693751 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.693725 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"cdac24c91b8bfb9688187aaef68eba4f4e4c0bc3452bf60aa6687219885cf893"} Oct 29 22:42:14.698895 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.698870 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lgqrb" event={"ID":"4fd11cc5-35ab-4347-8914-e953b71bc322","Type":"ContainerStarted","Data":"01a49e9fa13dadb93adfb64727b600a7c5da3af980482e09cc09f40155e3186a"} Oct 29 22:42:14.703344 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.703315 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" event={"ID":"f3d32c15ddf0c3d5f6df2743c0d67b79","Type":"ContainerStarted","Data":"5253c13d578973165ea4728d14e83bc389dd35f206d21f869bdf1eddc93b1b0b"} Oct 29 22:42:14.711660 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.711500 2575 generic.go:356] "Generic (PLEG): container finished" podID="3d509ddddb3f91e54ef73594cf479892" containerID="7dd5c1c7c273402203bb71f30cb821b9395848265324e78f4b15dd78b225ce09" exitCode=0 Oct 29 22:42:14.711660 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.711597 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" event={"ID":"3d509ddddb3f91e54ef73594cf479892","Type":"ContainerDied","Data":"7dd5c1c7c273402203bb71f30cb821b9395848265324e78f4b15dd78b225ce09"} Oct 29 22:42:14.720813 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.720729 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cpp5c" event={"ID":"6a6a57a5-af08-4ceb-a886-c176359b8d5a","Type":"ContainerStarted","Data":"85b7a4c9fffe7efecd43ab2ccecb1c9b8ee8b0432a9c97d0246b279322296d09"} Oct 29 22:42:14.722766 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.722739 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5rgg6" event={"ID":"8b726c8b-651c-4d5e-afa5-26a9f159d609","Type":"ContainerStarted","Data":"f4c14097f29a0b32410b03debe8ffa4f099aa6ee1d7b888c4c32ac2ba978f410"} Oct 29 22:42:14.747124 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:14.747066 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" podStartSLOduration=2.747049132 podStartE2EDuration="2.747049132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:14.746704201 +0000 UTC m=+3.707570798" watchObservedRunningTime="2026-10-29 22:42:14.747049132 +0000 UTC m=+3.707915731" Oct 29 22:42:15.122192 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:15.122139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:15.122192 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:15.122195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:15.122813 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.122792 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:15.122914 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.122868 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs podName:e85effca-b850-4af4-9086-ccfd965b7934 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:17.122847751 +0000 UTC m=+6.083714346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs") pod "network-metrics-daemon-d286w" (UID: "e85effca-b850-4af4-9086-ccfd965b7934") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:15.123323 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.123252 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:15.123323 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.123306 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret podName:fb913afd-66ca-46e3-b9d1-66cff498245d nodeName:}" failed. No retries permitted until 2026-10-29 22:42:17.123291232 +0000 UTC m=+6.084157806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret") pod "global-pull-secret-syncer-twb2h" (UID: "fb913afd-66ca-46e3-b9d1-66cff498245d") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:15.223846 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:15.223809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:15.224016 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.223957 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:15.224016 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.223971 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:15.224016 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.223980 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2skn4 for pod openshift-network-diagnostics/network-check-target-lglpm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:15.224176 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.224022 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4 podName:7d4c20d8-f5e8-450b-b4f4-8199b277bbb0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:17.224009292 +0000 UTC m=+6.184875867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2skn4" (UniqueName: "kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4") pod "network-check-target-lglpm" (UID: "7d4c20d8-f5e8-450b-b4f4-8199b277bbb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:15.463486 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:15.463439 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:37:13 +0000 UTC" deadline="2028-08-08 13:05:34.84656409 +0000 UTC" Oct 29 22:42:15.463486 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:15.463478 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15566h23m19.383090853s" Oct 29 22:42:15.643196 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:15.643170 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:15.643367 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.643295 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:15.644131 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:15.643515 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:15.644131 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:15.643681 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:15.735817 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:15.735619 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" event={"ID":"3d509ddddb3f91e54ef73594cf479892","Type":"ContainerStarted","Data":"4477b3bfe077fba351aa56281f42131d92051d515246b0e0fef732ff9e52c4d7"} Oct 29 22:42:16.638248 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:16.638217 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:16.638731 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:16.638354 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:16.742417 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:16.742380 2575 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="2df887541f355dde6d0f9978107aafa687f65bf18541766b29d1551cf15c1121" exitCode=0 Oct 29 22:42:16.743480 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:16.743453 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" event={"ID":"b78e901b-2f4b-4ee5-a96d-573988c5cadd","Type":"ContainerDied","Data":"2df887541f355dde6d0f9978107aafa687f65bf18541766b29d1551cf15c1121"} Oct 29 22:42:16.829282 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:16.829226 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" podStartSLOduration=4.829206676 podStartE2EDuration="4.829206676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:15.779632247 +0000 UTC m=+4.740498844" watchObservedRunningTime="2026-10-29 22:42:16.829206676 +0000 UTC m=+5.790073275" Oct 29 22:42:17.140541 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:17.140406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:17.140541 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:17.140457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:17.140796 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.140573 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:17.140796 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.140649 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs podName:e85effca-b850-4af4-9086-ccfd965b7934 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:21.14063088 +0000 UTC m=+10.101497469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs") pod "network-metrics-daemon-d286w" (UID: "e85effca-b850-4af4-9086-ccfd965b7934") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:17.141073 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.141055 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:17.141165 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.141117 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret podName:fb913afd-66ca-46e3-b9d1-66cff498245d nodeName:}" failed. No retries permitted until 2026-10-29 22:42:21.141095008 +0000 UTC m=+10.101961599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret") pod "global-pull-secret-syncer-twb2h" (UID: "fb913afd-66ca-46e3-b9d1-66cff498245d") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:17.241184 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:17.241152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:17.241339 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.241313 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:17.241339 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.241330 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:17.241443 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.241343 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2skn4 for pod openshift-network-diagnostics/network-check-target-lglpm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:17.241443 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.241393 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4 podName:7d4c20d8-f5e8-450b-b4f4-8199b277bbb0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:21.241376664 +0000 UTC m=+10.202243259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2skn4" (UniqueName: "kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4") pod "network-check-target-lglpm" (UID: "7d4c20d8-f5e8-450b-b4f4-8199b277bbb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:17.640676 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:17.640064 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:17.640676 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.640189 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:17.640676 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:17.640552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:17.642107 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:17.641983 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:17.752928 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:17.752192 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lgqrb" event={"ID":"4fd11cc5-35ab-4347-8914-e953b71bc322","Type":"ContainerStarted","Data":"b5017da51cc7c2a6983065639d5e2f4d59da3061d67043a5669e7df73dfbcdf5"} Oct 29 22:42:17.797720 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:17.796789 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lgqrb" podStartSLOduration=3.334515013 podStartE2EDuration="6.796769564s" totalImagesPullingTime="3.462254551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:17.796409289 +0000 UTC m=+6.757275887" watchObservedRunningTime="2026-10-29 22:42:17.796769564 +0000 UTC m=+6.757636164" Oct 29 22:42:18.637987 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:18.637902 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:18.638152 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:18.638107 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:19.639943 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:19.639871 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:19.640295 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:19.640000 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:19.640389 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:19.640372 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:19.640497 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:19.640470 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:19.757409 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:19.757372 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" event={"ID":"a66d8310-cc78-4a48-bc21-03fa742d0829","Type":"ContainerStarted","Data":"7cce2a98fcf8b85fc018bbef71c1c7e2cbb21f14525d0fd23ca101f724168469"} Oct 29 22:42:19.761443 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:19.761411 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ncjz6" event={"ID":"878dc633-6f09-450f-ab79-8788f14e3055","Type":"ContainerStarted","Data":"f6fc7106f69aff9ec8d13cf7c43b405b92e212fe847954f58668c9cad706ed80"} Oct 29 22:42:20.506007 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:20.505704 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:42:20.606300 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:20.605656 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:42:20.505725827Z","UUID":"d11d9fbb-a6c4-4029-a3e7-8a6a6549ca73","Handler":null,"Name":"","Endpoint":""} Oct 29 22:42:20.607322 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:20.607300 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:42:20.607435 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:20.607332 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:42:20.638025 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:20.637985 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:20.638155 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:20.638121 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:20.657895 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:20.657865 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:20.658589 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:20.658556 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:20.703985 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:20.702739 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ncjz6" podStartSLOduration=4.797429471 podStartE2EDuration="9.702719377s" totalImagesPullingTime="4.905289906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:19.806408082 +0000 UTC m=+8.767274679" watchObservedRunningTime="2026-10-29 22:42:20.702719377 +0000 UTC m=+9.663585975" Oct 29 22:42:20.766842 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:20.766342 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" event={"ID":"a66d8310-cc78-4a48-bc21-03fa742d0829","Type":"ContainerStarted","Data":"426a8d3ea20132bdf191f0ce12f23da9f9d0214bc4a59f0c90e3797f59dd2f1e"} Oct 29 22:42:21.175507 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:21.174813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:21.175507 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:21.174861 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:21.175507 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.174982 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:21.175507 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.175040 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs podName:e85effca-b850-4af4-9086-ccfd965b7934 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.175021451 +0000 UTC m=+18.135888039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs") pod "network-metrics-daemon-d286w" (UID: "e85effca-b850-4af4-9086-ccfd965b7934") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:21.175507 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.175426 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:21.175507 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.175475 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret podName:fb913afd-66ca-46e3-b9d1-66cff498245d nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.17546092 +0000 UTC m=+18.136327498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret") pod "global-pull-secret-syncer-twb2h" (UID: "fb913afd-66ca-46e3-b9d1-66cff498245d") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:21.275976 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:21.275926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:21.276116 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.276067 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:21.276116 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.276082 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:21.276116 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.276092 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2skn4 for pod openshift-network-diagnostics/network-check-target-lglpm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:21.276252 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.276141 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4 podName:7d4c20d8-f5e8-450b-b4f4-8199b277bbb0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:29.276121623 +0000 UTC m=+18.236988198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2skn4" (UniqueName: "kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4") pod "network-check-target-lglpm" (UID: "7d4c20d8-f5e8-450b-b4f4-8199b277bbb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:21.638911 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:21.638696 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:21.639528 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.639006 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:21.639528 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:21.639400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:21.639528 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:21.639492 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:21.770268 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:21.770234 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cpp5c" event={"ID":"6a6a57a5-af08-4ceb-a886-c176359b8d5a","Type":"ContainerStarted","Data":"cceefa528ba921b7bfd9219e6da157c874bed6593d1096c1e8d87acf0f73a941"} Oct 29 22:42:21.773787 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:21.773169 2575 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="7235dd1607ddeb64c05e2b32c5977006fcbb1cfde64e44efbd7db4bd73fc8132" exitCode=0 Oct 29 22:42:21.773787 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:21.773218 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" event={"ID":"b78e901b-2f4b-4ee5-a96d-573988c5cadd","Type":"ContainerDied","Data":"7235dd1607ddeb64c05e2b32c5977006fcbb1cfde64e44efbd7db4bd73fc8132"} Oct 29 22:42:21.833030 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:21.832974 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cpp5c" podStartSLOduration=3.270244528 podStartE2EDuration="10.832956561s" totalImagesPullingTime="7.562712033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:21.832487968 +0000 UTC m=+10.793354570" watchObservedRunningTime="2026-10-29 22:42:21.832956561 +0000 UTC m=+10.793823162" Oct 29 22:42:22.638165 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:22.638129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:22.638331 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:22.638281 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:22.776948 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:22.776910 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5rgg6" event={"ID":"8b726c8b-651c-4d5e-afa5-26a9f159d609","Type":"ContainerStarted","Data":"c341663e6b54a1a409897fb8a776bbaff289b027ef8ebe5a419692d3b361ac3b"} Oct 29 22:42:22.780563 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:22.780529 2575 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="bfcf41f73e50404dab4980551ee4b5fc0d445cbeeef2d7169fdd5b1b04fe0f70" exitCode=0 Oct 29 22:42:22.780695 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:22.780611 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" event={"ID":"b78e901b-2f4b-4ee5-a96d-573988c5cadd","Type":"ContainerDied","Data":"bfcf41f73e50404dab4980551ee4b5fc0d445cbeeef2d7169fdd5b1b04fe0f70"} Oct 29 22:42:22.784712 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:22.784688 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" event={"ID":"a66d8310-cc78-4a48-bc21-03fa742d0829","Type":"ContainerStarted","Data":"3ce1f6e768e15ba96bb21794358c9dcfd30be71c889dca481b8992418f8d91b2"} Oct 29 22:42:22.821301 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:22.821248 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5rgg6" podStartSLOduration=3.912863122 podStartE2EDuration="11.821231942s" totalImagesPullingTime="7.90836882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:22.820664338 +0000 UTC m=+11.781530937" watchObservedRunningTime="2026-10-29 22:42:22.821231942 +0000 UTC m=+11.782098541" Oct 29 22:42:22.924571 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:22.924514 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" podStartSLOduration=4.356756601 podStartE2EDuration="11.924499135s" totalImagesPullingTime="7.567742534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:22.867695828 +0000 UTC m=+11.828562429" watchObservedRunningTime="2026-10-29 22:42:22.924499135 +0000 UTC m=+11.885365732" Oct 29 22:42:23.638678 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:23.638612 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:23.638678 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:23.638657 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:23.638861 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:23.638761 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:23.638861 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:23.638843 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:23.789048 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:23.789012 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jk6th" event={"ID":"f0087fb0-f6d1-4f82-b78b-7f460f997ef0","Type":"ContainerStarted","Data":"ebe7bf0a61f9472d9f42d12ab04ea096944a1ea904b3286e20824247c2ce2ccb"} Oct 29 22:42:23.790685 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:23.790654 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dcjkg" event={"ID":"54a953c6-9194-404e-b8af-7ef3540d0e6d","Type":"ContainerStarted","Data":"36b995fbc0ccf7911090c3a6ed75a2a0521f208d478fc471de478e696a3e47e6"} Oct 29 22:42:23.793542 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:23.793517 2575 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="8ef08b5be091be9291f84caeb6229589864058ed2cf784a84b8575d083b279a2" exitCode=0 Oct 29 22:42:23.793716 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:23.793615 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" event={"ID":"b78e901b-2f4b-4ee5-a96d-573988c5cadd","Type":"ContainerDied","Data":"8ef08b5be091be9291f84caeb6229589864058ed2cf784a84b8575d083b279a2"} Oct 29 22:42:23.843486 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:23.843447 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jk6th" podStartSLOduration=3.221498935 podStartE2EDuration="12.843434278s" totalImagesPullingTime="9.621935343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:23.842945493 +0000 UTC m=+12.803812098" watchObservedRunningTime="2026-10-29 22:42:23.843434278 +0000 UTC m=+12.804300874" Oct 29 22:42:23.937822 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:23.937741 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dcjkg" podStartSLOduration=5.196605684 podStartE2EDuration="12.93772853s" totalImagesPullingTime="7.741122846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:23.937303406 +0000 UTC m=+12.898170001" watchObservedRunningTime="2026-10-29 22:42:23.93772853 +0000 UTC m=+12.898595127" Oct 29 22:42:24.637891 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:24.637864 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:24.638061 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:24.637990 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:25.637840 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:25.637765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:25.638356 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:25.637900 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:25.638356 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:25.637960 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:25.638356 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:25.638083 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:25.801537 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:25.801344 2575 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="2a1ff3d8868ee7ec52f73f6554071db48e3f1fcb1b708d42536809688f509997" exitCode=0 Oct 29 22:42:25.801537 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:25.801473 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" event={"ID":"b78e901b-2f4b-4ee5-a96d-573988c5cadd","Type":"ContainerDied","Data":"2a1ff3d8868ee7ec52f73f6554071db48e3f1fcb1b708d42536809688f509997"} Oct 29 22:42:26.638407 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:26.638378 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:26.638901 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:26.638485 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:26.807648 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:26.807535 2575 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="439b8e91ad01d997eaf08639c131b68e5a1d1a8bd5d35ff197e11dcfae24e298" exitCode=0 Oct 29 22:42:26.807648 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:26.807603 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" event={"ID":"b78e901b-2f4b-4ee5-a96d-573988c5cadd","Type":"ContainerDied","Data":"439b8e91ad01d997eaf08639c131b68e5a1d1a8bd5d35ff197e11dcfae24e298"} Oct 29 22:42:27.637805 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:27.637774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:27.637946 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:27.637820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:27.637946 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:27.637920 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:27.638060 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:27.638036 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:27.813654 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:27.813615 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" event={"ID":"b78e901b-2f4b-4ee5-a96d-573988c5cadd","Type":"ContainerStarted","Data":"66f037769b8284c1560139447449a9b0963162179b8340f34fda7d515a63cf54"} Oct 29 22:42:27.816888 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:27.815379 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"02a48f5861a7305427c0335950a2bfe7a55381b3582c1d7e83233968a255f7c1"} Oct 29 22:42:27.816888 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:27.815436 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"33602215d8aea52b31a320d1d01da54af469707127e7def266d9f62c109a0c71"} Oct 29 22:42:27.896302 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:27.896174 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ptfk2" podStartSLOduration=7.167034613 podStartE2EDuration="16.896161063s" totalImagesPullingTime="9.72912645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:27.895940492 +0000 UTC m=+16.856807089" watchObservedRunningTime="2026-10-29 22:42:27.896161063 +0000 UTC m=+16.857027660" Oct 29 22:42:28.637689 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:28.637660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:28.637826 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:28.637773 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:28.819955 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:28.819928 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:42:28.820501 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:28.820224 2575 generic.go:356] "Generic (PLEG): container finished" podID="d839fc45-f2f4-4ce4-95d8-9e3145342973" containerID="02a48f5861a7305427c0335950a2bfe7a55381b3582c1d7e83233968a255f7c1" exitCode=1 Oct 29 22:42:28.820501 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:28.820291 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerDied","Data":"02a48f5861a7305427c0335950a2bfe7a55381b3582c1d7e83233968a255f7c1"} Oct 29 22:42:28.820501 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:28.820363 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"96528e91b7876fbca3855d988f0a36ea83ed530cfd1fbd71d8ce0583e83d33ca"} Oct 29 22:42:28.820501 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:28.820409 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"f6929a96f8294e368bf387778dc66923bddd061bfd3d823c0b39f7d946f63533"} Oct 29 22:42:28.820501 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:28.820456 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"65e63b19cfbc8acf31f4e18bee3b60abf1e116ffdf533e63477f456587ff87d0"} Oct 29 22:42:28.820682 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:28.820505 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"82c58a5c568f3c44d3b5e7c12ef22c99e93889126da9bee64262c1beb060295f"} Oct 29 22:42:29.240255 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:29.240224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:29.240255 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:29.240262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:29.240452 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.240366 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:29.240452 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.240420 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret podName:fb913afd-66ca-46e3-b9d1-66cff498245d nodeName:}" failed. No retries permitted until 2026-10-29 22:42:45.240407593 +0000 UTC m=+34.201274172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret") pod "global-pull-secret-syncer-twb2h" (UID: "fb913afd-66ca-46e3-b9d1-66cff498245d") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:29.240529 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.240366 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:29.240529 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.240495 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs podName:e85effca-b850-4af4-9086-ccfd965b7934 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:45.240484057 +0000 UTC m=+34.201350647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs") pod "network-metrics-daemon-d286w" (UID: "e85effca-b850-4af4-9086-ccfd965b7934") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:29.340665 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:29.340633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:29.340812 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.340751 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:29.340812 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.340766 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:29.340812 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.340775 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2skn4 for pod openshift-network-diagnostics/network-check-target-lglpm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:29.340812 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.340813 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4 podName:7d4c20d8-f5e8-450b-b4f4-8199b277bbb0 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:45.34080192 +0000 UTC m=+34.301668495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2skn4" (UniqueName: "kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4") pod "network-check-target-lglpm" (UID: "7d4c20d8-f5e8-450b-b4f4-8199b277bbb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:29.637897 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:29.637815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:29.637897 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:29.637850 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:29.638104 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.637928 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:29.638104 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:29.638048 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:30.637981 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:30.637910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:30.638351 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:30.638013 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:30.828222 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:30.828182 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:42:30.828955 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:30.828922 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"69d868b3740de2e9b21bab5cb1e6bfe8e579652f09d92073c5f96adaacebe841"} Oct 29 22:42:31.640020 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:31.639993 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:31.640441 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:31.640106 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:31.640441 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:31.640137 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:31.640441 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:31.640188 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:32.638777 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:32.638517 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:32.638911 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:32.638795 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:32.834888 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:32.834862 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:42:32.835337 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:32.835199 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"0aecf3936fc20acad058d8ec3c105c96e390905adf18458d9ea478f849ab98fe"} Oct 29 22:42:32.835554 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:32.835536 2575 scope.go:122] "RemoveContainer" containerID="02a48f5861a7305427c0335950a2bfe7a55381b3582c1d7e83233968a255f7c1" Oct 29 22:42:32.835554 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:32.835550 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:32.835730 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:32.835620 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:32.851094 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:32.851076 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:33.457313 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.457288 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5rgg6_8b726c8b-651c-4d5e-afa5-26a9f159d609/dns-node-resolver/0.log" Oct 29 22:42:33.638471 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.638444 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:33.638635 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.638564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:33.638635 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:33.638565 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:33.638722 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:33.638663 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:33.840523 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.840495 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:42:33.840915 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.840845 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" event={"ID":"d839fc45-f2f4-4ce4-95d8-9e3145342973","Type":"ContainerStarted","Data":"7c39308d2f8b6a91610988b6f83a7e335773a1535891200cc8afa830ba4700e9"} Oct 29 22:42:33.841121 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.841104 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:33.855675 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.855654 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:42:33.939013 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.938974 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" podStartSLOduration=9.104573934 podStartE2EDuration="22.938962856s" totalImagesPullingTime="13.834388922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:33.93742187 +0000 UTC m=+22.898288467" watchObservedRunningTime="2026-10-29 22:42:33.938962856 +0000 UTC m=+22.899829450" Oct 29 22:42:33.995833 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.995811 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:33.995941 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.995901 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:33.996364 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:33.996348 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:42:34.386576 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:34.386553 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ncjz6_878dc633-6f09-450f-ab79-8788f14e3055/node-ca/0.log" Oct 29 22:42:34.638129 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:34.638062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:34.638264 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:34.638161 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:35.638063 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:35.638033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:35.638639 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:35.638197 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:35.638639 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:35.638235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:35.638639 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:35.638311 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:36.638478 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:36.638452 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:36.638859 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:36.638552 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:37.637851 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:37.637819 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:37.637996 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:37.637819 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:37.637996 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:37.637924 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:37.637996 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:37.637986 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:38.637890 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:38.637858 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:38.638326 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:38.637956 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:39.638171 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:39.638144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:39.638171 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:39.638163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:39.638609 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:39.638240 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:39.638609 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:39.638356 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:40.638526 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:40.638497 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:40.639074 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:40.638648 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:41.638843 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:41.638813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:41.639240 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:41.638924 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:41.639240 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:41.638970 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:41.639240 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:41.639048 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:42.638087 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:42.638060 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:42.638243 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:42.638160 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:43.637856 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:43.637803 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:43.638344 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:43.637870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:43.638344 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:43.637985 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:43.638344 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:43.638159 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:44.637872 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:44.637844 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:44.638258 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:44.637943 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:45.265554 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:45.265521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:45.265554 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:45.265556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:45.265831 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.265699 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:45.265831 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.265736 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:45.265831 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.265769 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret podName:fb913afd-66ca-46e3-b9d1-66cff498245d nodeName:}" failed. No retries permitted until 2026-10-29 22:43:17.265749312 +0000 UTC m=+66.226615886 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret") pod "global-pull-secret-syncer-twb2h" (UID: "fb913afd-66ca-46e3-b9d1-66cff498245d") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:45.265831 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.265790 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs podName:e85effca-b850-4af4-9086-ccfd965b7934 nodeName:}" failed. No retries permitted until 2026-10-29 22:43:17.265779116 +0000 UTC m=+66.226645705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs") pod "network-metrics-daemon-d286w" (UID: "e85effca-b850-4af4-9086-ccfd965b7934") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:45.366574 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:45.366548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:45.366708 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.366694 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:45.366746 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.366709 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:45.366746 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.366721 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2skn4 for pod openshift-network-diagnostics/network-check-target-lglpm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:45.366810 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.366764 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4 podName:7d4c20d8-f5e8-450b-b4f4-8199b277bbb0 nodeName:}" failed. No retries permitted until 2026-10-29 22:43:17.366752409 +0000 UTC m=+66.327618985 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2skn4" (UniqueName: "kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4") pod "network-check-target-lglpm" (UID: "7d4c20d8-f5e8-450b-b4f4-8199b277bbb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:45.637939 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:45.637870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:45.637939 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:45.637902 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:45.638356 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.637977 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:45.638356 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:45.638017 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:46.638170 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:46.638142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:46.638550 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:46.638242 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:47.638285 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:47.638256 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:47.638285 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:47.638269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:47.638705 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:47.638352 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:47.638705 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:47.638480 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:48.637783 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:48.637756 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:48.637932 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:48.637865 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:49.451423 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:49.451174 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lglpm"] Oct 29 22:42:49.451875 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:49.451535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:49.451875 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:49.451643 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:49.455224 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:49.455198 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-twb2h"] Oct 29 22:42:49.455352 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:49.455258 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d286w"] Oct 29 22:42:49.455352 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:49.455329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:49.455457 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:49.455430 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:49.455611 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:49.455598 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:49.455687 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:49.455663 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:51.642103 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:51.640574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:51.642103 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:51.640762 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:51.642103 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:51.640822 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:51.642103 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:51.640953 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:51.642103 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:51.640989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:51.642103 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:51.641109 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:53.637970 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.637929 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:53.637970 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.637975 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:53.638480 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:53.638066 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-twb2h" podUID="fb913afd-66ca-46e3-b9d1-66cff498245d" Oct 29 22:42:53.638480 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:53.638118 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d286w" podUID="e85effca-b850-4af4-9086-ccfd965b7934" Oct 29 22:42:53.638480 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.638142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:53.638480 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:53.638220 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lglpm" podUID="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" Oct 29 22:42:53.845898 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.845874 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeReady" Oct 29 22:42:53.846003 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.845991 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 22:42:53.910917 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.910888 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q7trl"] Oct 29 22:42:53.931061 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.931038 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zcdnt"] Oct 29 22:42:53.931173 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.931155 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:53.934721 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.934704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d31f1eee-73b6-4b6d-bde0-1006646b0173-metrics-tls\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:53.934817 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.934750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d31f1eee-73b6-4b6d-bde0-1006646b0173-tmp-dir\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:53.934855 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.934819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qt87\" (UniqueName: \"kubernetes.io/projected/d31f1eee-73b6-4b6d-bde0-1006646b0173-kube-api-access-9qt87\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:53.934889 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.934856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d31f1eee-73b6-4b6d-bde0-1006646b0173-config-volume\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:53.947098 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.947078 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q7trl"] Oct 29 22:42:53.947183 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.947141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:53.950306 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.950287 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zcdnt"] Oct 29 22:42:53.952834 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.952813 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tvxzw"] Oct 29 22:42:53.978035 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.978017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:42:53.979137 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:53.979118 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tvxzw"] Oct 29 22:42:54.034475 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.034410 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:42:54.034628 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.034566 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:42:54.034734 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.034716 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:42:54.034791 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.034732 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:42:54.034865 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.034851 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:42:54.034920 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.034875 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:42:54.035064 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d31f1eee-73b6-4b6d-bde0-1006646b0173-metrics-tls\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:54.035123 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08b826d-232a-48be-ade2-dd833d8c3423-cert\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:42:54.035177 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrlvp\" (UniqueName: \"kubernetes.io/projected/b08b826d-232a-48be-ade2-dd833d8c3423-kube-api-access-rrlvp\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:42:54.035177 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035163 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cee899d2-5ec5-443d-b004-89ce10507f7d-data-volume\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.035282 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.035335 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xcj\" (UniqueName: \"kubernetes.io/projected/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-api-access-g8xcj\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.035394 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035377 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d31f1eee-73b6-4b6d-bde0-1006646b0173-tmp-dir\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:54.035480 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qt87\" (UniqueName: \"kubernetes.io/projected/d31f1eee-73b6-4b6d-bde0-1006646b0173-kube-api-access-9qt87\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:54.035542 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cee899d2-5ec5-443d-b004-89ce10507f7d-crio-socket\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.035628 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d31f1eee-73b6-4b6d-bde0-1006646b0173-config-volume\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:54.035628 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cee899d2-5ec5-443d-b004-89ce10507f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.035728 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.035642 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d31f1eee-73b6-4b6d-bde0-1006646b0173-tmp-dir\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:54.036062 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.036039 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d31f1eee-73b6-4b6d-bde0-1006646b0173-config-volume\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:54.039070 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.039051 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d31f1eee-73b6-4b6d-bde0-1006646b0173-metrics-tls\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:54.048360 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.048341 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nxz4d\"" Oct 29 22:42:54.048436 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.048384 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:42:54.048552 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.048537 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:42:54.048630 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.048543 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:42:54.048630 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.048612 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d6hqb\"" Oct 29 22:42:54.048734 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.048698 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vllbq\"" Oct 29 22:42:54.061142 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.061124 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qt87\" (UniqueName: \"kubernetes.io/projected/d31f1eee-73b6-4b6d-bde0-1006646b0173-kube-api-access-9qt87\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:54.136414 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrlvp\" (UniqueName: \"kubernetes.io/projected/b08b826d-232a-48be-ade2-dd833d8c3423-kube-api-access-rrlvp\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:42:54.136495 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cee899d2-5ec5-443d-b004-89ce10507f7d-data-volume\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.136495 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.136495 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xcj\" (UniqueName: \"kubernetes.io/projected/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-api-access-g8xcj\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.136635 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cee899d2-5ec5-443d-b004-89ce10507f7d-crio-socket\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.136676 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cee899d2-5ec5-443d-b004-89ce10507f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.136710 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08b826d-232a-48be-ade2-dd833d8c3423-cert\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:42:54.136739 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cee899d2-5ec5-443d-b004-89ce10507f7d-crio-socket\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.136850 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136832 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cee899d2-5ec5-443d-b004-89ce10507f7d-data-volume\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.136972 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.136956 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.138822 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.138799 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08b826d-232a-48be-ade2-dd833d8c3423-cert\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:42:54.138896 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.138805 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cee899d2-5ec5-443d-b004-89ce10507f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.162184 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.162162 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xcj\" (UniqueName: \"kubernetes.io/projected/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-api-access-g8xcj\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.162265 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.162191 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrlvp\" (UniqueName: \"kubernetes.io/projected/b08b826d-232a-48be-ade2-dd833d8c3423-kube-api-access-rrlvp\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:42:54.266338 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.266309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:54.298686 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.298633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:42:54.307104 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.307064 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:42:54.572790 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.572769 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q7trl"] Oct 29 22:42:54.575441 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:54.575420 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31f1eee_73b6_4b6d_bde0_1006646b0173.slice/crio-bafb3bf21c841d6aa1a98ea0c2751e1932d384abf57dd6c6ea8f2a6f400b4d3e WatchSource:0}: Error finding container bafb3bf21c841d6aa1a98ea0c2751e1932d384abf57dd6c6ea8f2a6f400b4d3e: Status 404 returned error can't find the container with id bafb3bf21c841d6aa1a98ea0c2751e1932d384abf57dd6c6ea8f2a6f400b4d3e Oct 29 22:42:54.603132 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.603113 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zcdnt"] Oct 29 22:42:54.605864 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:54.605843 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee899d2_5ec5_443d_b004_89ce10507f7d.slice/crio-e6d7fa437c70919c1cce9fcd1945c52472980b7628e5ce4e09cebd5c9d5cecfa WatchSource:0}: Error finding container e6d7fa437c70919c1cce9fcd1945c52472980b7628e5ce4e09cebd5c9d5cecfa: Status 404 returned error can't find the container with id e6d7fa437c70919c1cce9fcd1945c52472980b7628e5ce4e09cebd5c9d5cecfa Oct 29 22:42:54.613356 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.613333 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tvxzw"] Oct 29 22:42:54.616076 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:54.616053 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08b826d_232a_48be_ade2_dd833d8c3423.slice/crio-912b8930f9f8f8e907be377a9381118ceccef6fe57cafc8ab865b5a67e2a3952 WatchSource:0}: Error finding container 912b8930f9f8f8e907be377a9381118ceccef6fe57cafc8ab865b5a67e2a3952: Status 404 returned error can't find the container with id 912b8930f9f8f8e907be377a9381118ceccef6fe57cafc8ab865b5a67e2a3952 Oct 29 22:42:54.883755 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.883662 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tvxzw" event={"ID":"b08b826d-232a-48be-ade2-dd833d8c3423","Type":"ContainerStarted","Data":"912b8930f9f8f8e907be377a9381118ceccef6fe57cafc8ab865b5a67e2a3952"} Oct 29 22:42:54.885308 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.885278 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zcdnt" event={"ID":"cee899d2-5ec5-443d-b004-89ce10507f7d","Type":"ContainerStarted","Data":"322c746396754c34472bc7e41b5936316741fbeb31b1155c0480e098c1aac3fe"} Oct 29 22:42:54.885420 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.885344 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zcdnt" event={"ID":"cee899d2-5ec5-443d-b004-89ce10507f7d","Type":"ContainerStarted","Data":"e6d7fa437c70919c1cce9fcd1945c52472980b7628e5ce4e09cebd5c9d5cecfa"} Oct 29 22:42:54.886443 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:54.886420 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7trl" event={"ID":"d31f1eee-73b6-4b6d-bde0-1006646b0173","Type":"ContainerStarted","Data":"bafb3bf21c841d6aa1a98ea0c2751e1932d384abf57dd6c6ea8f2a6f400b4d3e"} Oct 29 22:42:55.638838 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.638034 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:42:55.638838 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.638145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:42:55.638838 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.638449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:42:55.670746 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.670442 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:42:55.670746 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.670464 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:42:55.670746 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.670654 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:42:55.670746 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.670680 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mcv8v\"" Oct 29 22:42:55.670746 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.670693 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:42:55.670746 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.670728 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ltz9s\"" Oct 29 22:42:55.671133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.670852 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kfq4\"" Oct 29 22:42:55.891673 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:55.891572 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zcdnt" event={"ID":"cee899d2-5ec5-443d-b004-89ce10507f7d","Type":"ContainerStarted","Data":"ca7ac07d7ec1da0a8e98b2dcbc0789ad0a54e78fa912238339a12e9bd7cf1b73"} Oct 29 22:42:56.155942 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.155863 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7"] Oct 29 22:42:56.158493 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.158465 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.183485 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.183460 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:42:56.183671 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.183540 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 29 22:42:56.184610 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.184589 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7"] Oct 29 22:42:56.197174 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.197150 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:42:56.197299 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.197245 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:42:56.197381 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.197305 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:42:56.197381 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.197325 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nqp65\"" Oct 29 22:42:56.198558 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.198538 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf"] Oct 29 22:42:56.201179 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.201156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.221819 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.221799 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rc9cg"] Oct 29 22:42:56.223753 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.223734 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.252662 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-root\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.252773 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252688 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.252773 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.252773 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-sys\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.252773 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58hz\" (UniqueName: \"kubernetes.io/projected/b1d25bfd-dba6-4268-8123-533f4b106c0a-kube-api-access-w58hz\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.252935 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252789 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.252935 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj97r\" (UniqueName: \"kubernetes.io/projected/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-kube-api-access-rj97r\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.252935 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-textfile\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.252935 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-metrics-client-ca\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.252935 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-tls\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.253150 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252938 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.253150 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.253150 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.252988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-wtmp\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.292081 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.292051 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:42:56.292230 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.292200 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 29 22:42:56.292350 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.292279 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ljgzz\"" Oct 29 22:42:56.292406 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.292397 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:42:56.292473 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.292454 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:42:56.292598 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.292544 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x4fl8\"" Oct 29 22:42:56.293257 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.293185 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:42:56.293338 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.293299 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 29 22:42:56.306687 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.306667 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf"] Oct 29 22:42:56.355729 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.355690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/875f6c48-91a9-4f0b-a71d-e41d02e589d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.355909 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.355755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.355909 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.355793 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.355909 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.355831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58hz\" (UniqueName: \"kubernetes.io/projected/b1d25bfd-dba6-4268-8123-533f4b106c0a-kube-api-access-w58hz\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.356362 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.356472 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-wtmp\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.356472 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.356698 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356567 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-wtmp\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.356698 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.356827 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4kmz\" (UniqueName: \"kubernetes.io/projected/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-api-access-p4kmz\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.356827 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-root\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.356827 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.356827 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-sys\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.357024 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.357024 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj97r\" (UniqueName: \"kubernetes.io/projected/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-kube-api-access-rj97r\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.357024 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.357024 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-textfile\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.357024 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.356988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-metrics-client-ca\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.357259 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.357029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-tls\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.357259 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.357061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.357259 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.357238 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.357785 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.357766 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-textfile\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.357914 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.357895 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-sys\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.357978 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.357966 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-root\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.358489 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.358187 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.358617 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.358597 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-metrics-client-ca\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.361433 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.361408 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.361825 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.361767 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.363521 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.363500 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-tls\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.370306 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.369864 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.383133 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.383108 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58hz\" (UniqueName: \"kubernetes.io/projected/b1d25bfd-dba6-4268-8123-533f4b106c0a-kube-api-access-w58hz\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.385312 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.385288 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj97r\" (UniqueName: \"kubernetes.io/projected/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-kube-api-access-rj97r\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.457777 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.457488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.457777 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.457537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.457777 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.457597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kmz\" (UniqueName: \"kubernetes.io/projected/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-api-access-p4kmz\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.457777 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.457634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.457777 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.457666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/875f6c48-91a9-4f0b-a71d-e41d02e589d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.457777 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.457697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.458336 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.458245 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/875f6c48-91a9-4f0b-a71d-e41d02e589d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.458445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.458335 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.458445 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.458382 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.460283 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.460261 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.460484 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.460456 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.474175 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.473997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:42:56.488305 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.488283 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kmz\" (UniqueName: \"kubernetes.io/projected/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-api-access-p4kmz\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.515168 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.515138 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:42:56.538802 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.538775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:42:56.801281 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.801162 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7"] Oct 29 22:42:56.803436 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:56.803407 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdf7ea9_aed7_4071_ada6_2f0518de3a87.slice/crio-3b7f9f9487f36b130419d5e13f19527462b0294d67d287abb2cd625b121036ef WatchSource:0}: Error finding container 3b7f9f9487f36b130419d5e13f19527462b0294d67d287abb2cd625b121036ef: Status 404 returned error can't find the container with id 3b7f9f9487f36b130419d5e13f19527462b0294d67d287abb2cd625b121036ef Oct 29 22:42:56.854401 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.852996 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf"] Oct 29 22:42:56.897310 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.897276 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7trl" event={"ID":"d31f1eee-73b6-4b6d-bde0-1006646b0173","Type":"ContainerStarted","Data":"d3f1620851395bde1fc5486749f2a81c4192b9d9fa9b016607e98ceb927ec73f"} Oct 29 22:42:56.897626 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.897343 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7trl" event={"ID":"d31f1eee-73b6-4b6d-bde0-1006646b0173","Type":"ContainerStarted","Data":"5a3834736ec32018639b78927c7fbd0615fe7a1496b31f2303efcffe2008284b"} Oct 29 22:42:56.897966 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.897946 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q7trl" Oct 29 22:42:56.899527 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.899460 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rc9cg" event={"ID":"b1d25bfd-dba6-4268-8123-533f4b106c0a","Type":"ContainerStarted","Data":"465e1464aa13f2b03e5119bd5dd6ad2cc127bc7a0a8b68d14ef7e01e3d5ebe0a"} Oct 29 22:42:56.902227 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.902205 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zcdnt" event={"ID":"cee899d2-5ec5-443d-b004-89ce10507f7d","Type":"ContainerStarted","Data":"c0bb53c89bdfd30e652a19347ce796b8e399c019f915b527f0904ecd7695db3d"} Oct 29 22:42:56.904560 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.904534 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" event={"ID":"8cdf7ea9-aed7-4071-ada6-2f0518de3a87","Type":"ContainerStarted","Data":"6d96aecb931d19e219e80e8e13c04da72a8d0797c1c2685f1ae70f75442f9352"} Oct 29 22:42:56.904664 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.904611 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" event={"ID":"8cdf7ea9-aed7-4071-ada6-2f0518de3a87","Type":"ContainerStarted","Data":"3b7f9f9487f36b130419d5e13f19527462b0294d67d287abb2cd625b121036ef"} Oct 29 22:42:56.907684 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.907630 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" event={"ID":"875f6c48-91a9-4f0b-a71d-e41d02e589d5","Type":"ContainerStarted","Data":"f092da451c9d7ac853e0529c3e2ed53ec04ddfc1e9918c52c86c293a711b6faf"} Oct 29 22:42:56.994893 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.994836 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zcdnt" podStartSLOduration=1.950631175 podStartE2EDuration="3.994821916s" totalImagesPullingTime="2.044190741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:56.994456806 +0000 UTC m=+45.955323404" watchObservedRunningTime="2026-10-29 22:42:56.994821916 +0000 UTC m=+45.955688513" Oct 29 22:42:56.995226 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:56.995199 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-q7trl" podStartSLOduration=2.200047825 podStartE2EDuration="3.99518615s" totalImagesPullingTime="1.795138325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:56.946543609 +0000 UTC m=+45.907410208" watchObservedRunningTime="2026-10-29 22:42:56.99518615 +0000 UTC m=+45.956052748" Oct 29 22:42:57.616188 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.614169 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:42:57.618262 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.617280 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.638325 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.638289 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 22:42:57.638477 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.638449 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 22:42:57.640610 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.639090 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 22:42:57.640610 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.639263 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 22:42:57.640610 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.639336 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-98j5z\"" Oct 29 22:42:57.644947 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.644891 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 22:42:57.652415 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.652348 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:42:57.669806 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.669770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.669932 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.669819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.669932 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.669872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-config-volume\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.669932 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.669900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-web-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.669932 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.669923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.670134 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.669968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.670134 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.669999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.670134 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.670033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-config-out\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.670134 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.670057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.670134 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.670082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.670134 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.670125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.670362 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.670156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.670362 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.670185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngpp\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-kube-api-access-fngpp\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.737903 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.737857 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 22:42:57.737903 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.737892 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 22:42:57.738071 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.737899 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 22:42:57.738071 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.738022 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 22:42:57.770702 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.770672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.770835 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.770725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.770835 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.770758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngpp\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-kube-api-access-fngpp\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.770835 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.770802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.770835 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.770833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.771351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.771080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-config-volume\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.771351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.771125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-web-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.771351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.771154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.771351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.771207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.771351 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.771238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.771622 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.771438 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.771622 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.771523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-config-out\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.771622 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:57.771563 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle podName:9eb7d909-e423-4efb-89ba-97cddc92628e nodeName:}" failed. No retries permitted until 2026-10-29 22:42:58.271544733 +0000 UTC m=+47.232411328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9eb7d909-e423-4efb-89ba-97cddc92628e") : configmap references non-existent config key: ca-bundle.crt Oct 29 22:42:57.771622 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.771601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.771772 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.771630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.772299 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.772265 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.779011 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.778990 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-config-out\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.779248 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.779225 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.779334 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.779251 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.779452 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.779427 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.780043 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.780009 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.780112 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.780069 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-config-volume\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.780112 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.780093 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-web-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.780190 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.780121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.781924 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.781882 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.798975 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.798938 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngpp\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-kube-api-access-fngpp\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:57.914536 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.914509 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" event={"ID":"8cdf7ea9-aed7-4071-ada6-2f0518de3a87","Type":"ContainerStarted","Data":"200722c31d340ab3fc3c37af9009cfade2adaa3ee2f203423840fa2ec7ddb792"} Oct 29 22:42:57.916413 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.916379 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tvxzw" event={"ID":"b08b826d-232a-48be-ade2-dd833d8c3423","Type":"ContainerStarted","Data":"aea72db44d7b6ccdc4f2fc59cd55d518e65ae39d64d8e4f2917922460b64378f"} Oct 29 22:42:57.918118 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.918087 2575 generic.go:356] "Generic (PLEG): container finished" podID="b1d25bfd-dba6-4268-8123-533f4b106c0a" containerID="641f9013df5c386a1f2fd590c4b60c309a1964ef2ff41458fc166b6e7a0c3251" exitCode=0 Oct 29 22:42:57.918327 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:57.918254 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rc9cg" event={"ID":"b1d25bfd-dba6-4268-8123-533f4b106c0a","Type":"ContainerDied","Data":"641f9013df5c386a1f2fd590c4b60c309a1964ef2ff41458fc166b6e7a0c3251"} Oct 29 22:42:58.014752 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.014696 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tvxzw" podStartSLOduration=1.939682524 podStartE2EDuration="5.014676724s" totalImagesPullingTime="3.0749942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:57.961453364 +0000 UTC m=+46.922319980" watchObservedRunningTime="2026-10-29 22:42:58.014676724 +0000 UTC m=+46.975543322" Oct 29 22:42:58.276844 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.276782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:58.276945 ip-10-0-143-172 kubenswrapper[2575]: E1029 22:42:58.276898 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle podName:9eb7d909-e423-4efb-89ba-97cddc92628e nodeName:}" failed. No retries permitted until 2026-10-29 22:42:59.276883982 +0000 UTC m=+48.237750569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9eb7d909-e423-4efb-89ba-97cddc92628e") : configmap references non-existent config key: ca-bundle.crt Oct 29 22:42:58.415642 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.415606 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86c4948c74-qwn7r"] Oct 29 22:42:58.432064 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.432037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.448206 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.448169 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86c4948c74-qwn7r"] Oct 29 22:42:58.450039 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.450008 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 29 22:42:58.450039 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.450023 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 29 22:42:58.450039 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.450010 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 29 22:42:58.450259 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.450091 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 29 22:42:58.450259 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.450138 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dtoi9vcgc088c\"" Oct 29 22:42:58.450259 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.450224 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xvchs\"" Oct 29 22:42:58.453811 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.453788 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 29 22:42:58.478668 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.478643 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.478755 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.478673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.478755 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.478704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.478880 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.478862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfnkq\" (UniqueName: \"kubernetes.io/projected/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-kube-api-access-dfnkq\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.478911 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.478902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.478939 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.478921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-metrics-client-ca\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.478968 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.478949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.479098 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.479077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-grpc-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.479131 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.479117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-tmp\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580375 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-tmp\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580375 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580375 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580609 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580609 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfnkq\" (UniqueName: \"kubernetes.io/projected/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-kube-api-access-dfnkq\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580609 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580758 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-metrics-client-ca\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580758 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580858 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-grpc-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.580858 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.580791 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-tmp\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.581472 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.581422 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-metrics-client-ca\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.583511 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.583482 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.583641 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.583545 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.583641 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.583572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.583829 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.583807 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-grpc-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.583952 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.583935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.584143 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.584124 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.607509 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.607484 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfnkq\" (UniqueName: \"kubernetes.io/projected/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-kube-api-access-dfnkq\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.753462 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.753432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:42:58.923870 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.923783 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rc9cg" event={"ID":"b1d25bfd-dba6-4268-8123-533f4b106c0a","Type":"ContainerStarted","Data":"3e575f6ecfa04606e78a173ed7816705be834a0e3bdb4766d4ec0c2335ffe819"} Oct 29 22:42:58.923870 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.923849 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rc9cg" event={"ID":"b1d25bfd-dba6-4268-8123-533f4b106c0a","Type":"ContainerStarted","Data":"5e398e75c0e85f21fbd82668c398b4de39835a8ada49899f478650de1305bcdd"} Oct 29 22:42:58.925866 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.925840 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" event={"ID":"8cdf7ea9-aed7-4071-ada6-2f0518de3a87","Type":"ContainerStarted","Data":"18cf44e67d8a4963af1cce1db4be3b950b36eefd148d356f9c1b14c37bdcaebc"} Oct 29 22:42:58.927976 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.927957 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" event={"ID":"875f6c48-91a9-4f0b-a71d-e41d02e589d5","Type":"ContainerStarted","Data":"b0466f6a3f8efb9691e822952674b59a162feb3f9b8de1bbae4c57b6882b1ebe"} Oct 29 22:42:58.928058 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.928000 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" event={"ID":"875f6c48-91a9-4f0b-a71d-e41d02e589d5","Type":"ContainerStarted","Data":"eaa1f52541e1ecbefc43044b7e9e79446e1730903d9f5d556eb4b02ac29f61f0"} Oct 29 22:42:58.928058 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.928030 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" event={"ID":"875f6c48-91a9-4f0b-a71d-e41d02e589d5","Type":"ContainerStarted","Data":"f5bfceacf1c632f0a6258150505c6231c7dda65c596b783966feae6032244098"} Oct 29 22:42:58.972110 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:58.972040 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rc9cg" podStartSLOduration=1.91639816 podStartE2EDuration="2.972025996s" totalImagesPullingTime="974.942043ms" totalInitContainerRuntime="80.685793ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:58.971488764 +0000 UTC m=+47.932355363" watchObservedRunningTime="2026-10-29 22:42:58.972025996 +0000 UTC m=+47.932892594" Oct 29 22:42:59.019110 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:59.019068 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" podStartSLOduration=1.779542951 podStartE2EDuration="3.019054172s" totalImagesPullingTime="1.239511221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:59.018362085 +0000 UTC m=+47.979228684" watchObservedRunningTime="2026-10-29 22:42:59.019054172 +0000 UTC m=+47.979920771" Oct 29 22:42:59.065724 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:59.065689 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" podStartSLOduration=2.058860366 podStartE2EDuration="3.065679267s" totalImagesPullingTime="1.006818901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:59.065557158 +0000 UTC m=+48.026423756" watchObservedRunningTime="2026-10-29 22:42:59.065679267 +0000 UTC m=+48.026545864" Oct 29 22:42:59.082902 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:59.082883 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86c4948c74-qwn7r"] Oct 29 22:42:59.084429 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:59.084403 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dc5151_8f55_47c4_a3a5_4e063c1e1ef5.slice/crio-d4fdf7c93e52fcc9844fa0399c8c18867ceed621cad347eac4551a8820553feb WatchSource:0}: Error finding container d4fdf7c93e52fcc9844fa0399c8c18867ceed621cad347eac4551a8820553feb: Status 404 returned error can't find the container with id d4fdf7c93e52fcc9844fa0399c8c18867ceed621cad347eac4551a8820553feb Oct 29 22:42:59.287461 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:59.287438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:59.288105 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:59.288085 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:59.454263 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:59.454231 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:42:59.814115 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:59.814087 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 22:42:59.815478 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:42:59.815447 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb7d909_e423_4efb_89ba_97cddc92628e.slice/crio-e3eb10ad2da597c1507fa4439f14ebc4837b39c346ee8ab6a1b1b48580b88395 WatchSource:0}: Error finding container e3eb10ad2da597c1507fa4439f14ebc4837b39c346ee8ab6a1b1b48580b88395: Status 404 returned error can't find the container with id e3eb10ad2da597c1507fa4439f14ebc4837b39c346ee8ab6a1b1b48580b88395 Oct 29 22:42:59.932542 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:59.932504 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9eb7d909-e423-4efb-89ba-97cddc92628e","Type":"ContainerStarted","Data":"e3eb10ad2da597c1507fa4439f14ebc4837b39c346ee8ab6a1b1b48580b88395"} Oct 29 22:42:59.933951 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:42:59.933914 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" event={"ID":"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5","Type":"ContainerStarted","Data":"d4fdf7c93e52fcc9844fa0399c8c18867ceed621cad347eac4551a8820553feb"} Oct 29 22:43:00.827403 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.827374 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-75f497d78f-wzp7l"] Oct 29 22:43:00.829272 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.829255 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:00.864958 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.864938 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 29 22:43:00.866003 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.865984 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s2zp5\"" Oct 29 22:43:00.866119 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.866094 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 29 22:43:00.866239 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.866138 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-62qeduce6lfgh\"" Oct 29 22:43:00.866239 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.866159 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 22:43:00.866239 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.866194 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 29 22:43:00.880856 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.880835 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75f497d78f-wzp7l"] Oct 29 22:43:00.902106 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.902056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-tls\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:00.902106 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.902097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:00.902347 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.902130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e4f37ae-5582-442e-bf04-a2cc29932ec6-audit-log\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:00.902347 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.902277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbwl\" (UniqueName: \"kubernetes.io/projected/1e4f37ae-5582-442e-bf04-a2cc29932ec6-kube-api-access-qqbwl\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:00.902347 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.902321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-client-certs\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:00.902540 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.902423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-metrics-server-audit-profiles\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:00.902540 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.902492 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-client-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:00.938711 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.938688 2575 generic.go:356] "Generic (PLEG): container finished" podID="9eb7d909-e423-4efb-89ba-97cddc92628e" containerID="184b92331e6d03d4aa342e3a323dc7bd191ee86dc6837d73fb89c21fa2a68e21" exitCode=0 Oct 29 22:43:00.939085 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.938777 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9eb7d909-e423-4efb-89ba-97cddc92628e","Type":"ContainerDied","Data":"184b92331e6d03d4aa342e3a323dc7bd191ee86dc6837d73fb89c21fa2a68e21"} Oct 29 22:43:00.941527 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.941505 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" event={"ID":"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5","Type":"ContainerStarted","Data":"d16f7e5d127eb0513aa51569d804e086f4581c6c8548495d5245fbd765cfc92a"} Oct 29 22:43:00.941656 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:00.941560 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" event={"ID":"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5","Type":"ContainerStarted","Data":"783895d0d514d2ada557029e4fc5ce212d8eacca242048fc574719a42731f315"} Oct 29 22:43:01.005748 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.005620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbwl\" (UniqueName: \"kubernetes.io/projected/1e4f37ae-5582-442e-bf04-a2cc29932ec6-kube-api-access-qqbwl\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.005748 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.005685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-client-certs\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.005748 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.005733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-metrics-server-audit-profiles\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.006055 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.005822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-client-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.006055 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.005945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-tls\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.006055 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.005983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.006055 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.006016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e4f37ae-5582-442e-bf04-a2cc29932ec6-audit-log\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.006993 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.006931 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e4f37ae-5582-442e-bf04-a2cc29932ec6-audit-log\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.007137 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.007111 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb"] Oct 29 22:43:01.007291 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.007270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.007869 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.007838 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-metrics-server-audit-profiles\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.008620 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.008601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-client-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.008856 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.008836 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-tls\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.008892 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.008836 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-client-certs\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.024034 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.024017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:01.036885 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.036847 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbwl\" (UniqueName: \"kubernetes.io/projected/1e4f37ae-5582-442e-bf04-a2cc29932ec6-kube-api-access-qqbwl\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.037155 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.037137 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb"] Oct 29 22:43:01.057244 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.057224 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f88xp\"" Oct 29 22:43:01.057349 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.057312 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 29 22:43:01.106392 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.106364 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb967bfd-fb93-4051-bda5-9dbdf3589a69-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:01.106516 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.106410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2gfd\" (UniqueName: \"kubernetes.io/projected/bb967bfd-fb93-4051-bda5-9dbdf3589a69-kube-api-access-w2gfd\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:01.139966 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.139906 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:01.207305 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.207271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2gfd\" (UniqueName: \"kubernetes.io/projected/bb967bfd-fb93-4051-bda5-9dbdf3589a69-kube-api-access-w2gfd\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:01.207428 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.207369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb967bfd-fb93-4051-bda5-9dbdf3589a69-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:01.209606 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.209569 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb967bfd-fb93-4051-bda5-9dbdf3589a69-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:01.235093 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.235066 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2gfd\" (UniqueName: \"kubernetes.io/projected/bb967bfd-fb93-4051-bda5-9dbdf3589a69-kube-api-access-w2gfd\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:01.335176 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.335140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:01.458639 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.458516 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75f497d78f-wzp7l"] Oct 29 22:43:01.462679 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:43:01.462647 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4f37ae_5582_442e_bf04_a2cc29932ec6.slice/crio-bb9f61588c747ca41785a29fa90332c8070870d825bc1ccca3e5ebba52f58702 WatchSource:0}: Error finding container bb9f61588c747ca41785a29fa90332c8070870d825bc1ccca3e5ebba52f58702: Status 404 returned error can't find the container with id bb9f61588c747ca41785a29fa90332c8070870d825bc1ccca3e5ebba52f58702 Oct 29 22:43:01.571513 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.571479 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76865c6696-5hm7s"] Oct 29 22:43:01.574089 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.574067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.595228 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.595200 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 29 22:43:01.610246 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.610128 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 29 22:43:01.610246 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.610165 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 29 22:43:01.610246 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.610185 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 29 22:43:01.610246 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.610211 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 29 22:43:01.610246 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.610238 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 29 22:43:01.610560 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.610268 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8xd28\"" Oct 29 22:43:01.610818 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.610792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.610933 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.610844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-metrics-client-ca\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.610993 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.610981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-serving-certs-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.611044 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.611008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqw6\" (UniqueName: \"kubernetes.io/projected/9009d30d-e890-4365-83af-2244164260f3-kube-api-access-wcqw6\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.611096 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.611073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.611149 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.611127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-federate-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.611225 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.611204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-telemeter-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.611340 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.611259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.611948 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.611929 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76865c6696-5hm7s"] Oct 29 22:43:01.650728 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.650659 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb"] Oct 29 22:43:01.653485 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:43:01.653453 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb967bfd_fb93_4051_bda5_9dbdf3589a69.slice/crio-49c4e62507582f24fda9808b6efcb0eb12d9cb9dedce54a2bc74930b1d423374 WatchSource:0}: Error finding container 49c4e62507582f24fda9808b6efcb0eb12d9cb9dedce54a2bc74930b1d423374: Status 404 returned error can't find the container with id 49c4e62507582f24fda9808b6efcb0eb12d9cb9dedce54a2bc74930b1d423374 Oct 29 22:43:01.712604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.712304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-serving-certs-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.712604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.712355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqw6\" (UniqueName: \"kubernetes.io/projected/9009d30d-e890-4365-83af-2244164260f3-kube-api-access-wcqw6\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.712604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.712383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.712604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.712405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-federate-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.712604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.712443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-telemeter-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.712604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.712469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.712604 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.712542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.713098 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.713065 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-serving-certs-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.713415 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.713383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-metrics-client-ca\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.713945 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.713922 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.714197 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.714171 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-metrics-client-ca\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.715355 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.715308 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.715501 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.715352 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-federate-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.715501 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.715384 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-telemeter-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.715501 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.715487 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.738991 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.738961 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqw6\" (UniqueName: \"kubernetes.io/projected/9009d30d-e890-4365-83af-2244164260f3-kube-api-access-wcqw6\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.888220 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.888200 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:01.967065 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.967030 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" event={"ID":"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5","Type":"ContainerStarted","Data":"17c589f6699db57dbbadca39f092d0e587c671e83410666b3cccf679760c1390"} Oct 29 22:43:01.967413 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.967154 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" event={"ID":"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5","Type":"ContainerStarted","Data":"6aefd9eb21666797ee47a3bdc8fa0831d1811d436b1adab697672a45730957c1"} Oct 29 22:43:01.967413 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.967204 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" event={"ID":"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5","Type":"ContainerStarted","Data":"793a2d402797b229bca5e266d68661ce4b15e1da7ad5808108f868c92af9b857"} Oct 29 22:43:01.969726 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.969680 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" event={"ID":"bb967bfd-fb93-4051-bda5-9dbdf3589a69","Type":"ContainerStarted","Data":"49c4e62507582f24fda9808b6efcb0eb12d9cb9dedce54a2bc74930b1d423374"} Oct 29 22:43:01.971720 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:01.971663 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" event={"ID":"1e4f37ae-5582-442e-bf04-a2cc29932ec6","Type":"ContainerStarted","Data":"bb9f61588c747ca41785a29fa90332c8070870d825bc1ccca3e5ebba52f58702"} Oct 29 22:43:02.239811 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.239719 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76865c6696-5hm7s"] Oct 29 22:43:02.245249 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:43:02.245216 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9009d30d_e890_4365_83af_2244164260f3.slice/crio-eb075e522fd531388bf83d138dc25235a9f448e3bf61996c12bf5ece6b62901c WatchSource:0}: Error finding container eb075e522fd531388bf83d138dc25235a9f448e3bf61996c12bf5ece6b62901c: Status 404 returned error can't find the container with id eb075e522fd531388bf83d138dc25235a9f448e3bf61996c12bf5ece6b62901c Oct 29 22:43:02.978816 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.978729 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" event={"ID":"9009d30d-e890-4365-83af-2244164260f3","Type":"ContainerStarted","Data":"eb075e522fd531388bf83d138dc25235a9f448e3bf61996c12bf5ece6b62901c"} Oct 29 22:43:02.984398 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.984366 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9eb7d909-e423-4efb-89ba-97cddc92628e","Type":"ContainerStarted","Data":"450a553dc74eae63ec037ff5990ee4fba80cf7ecd5ca82d229686cb6b6f29841"} Oct 29 22:43:02.984511 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.984444 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9eb7d909-e423-4efb-89ba-97cddc92628e","Type":"ContainerStarted","Data":"384db7dd2a0fd29ae023c2b626626572f10902ddbe9e73e03394a58503491865"} Oct 29 22:43:02.984511 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.984483 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9eb7d909-e423-4efb-89ba-97cddc92628e","Type":"ContainerStarted","Data":"3cfe801848611f7fd7a46aee3e480fc5a1f669f1613443d7fec1916473c33841"} Oct 29 22:43:02.984611 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.984520 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9eb7d909-e423-4efb-89ba-97cddc92628e","Type":"ContainerStarted","Data":"fb530337329d7827fcaf0d395870c791eebf5612d2206fb7b926d98303f8444a"} Oct 29 22:43:02.984611 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.984558 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9eb7d909-e423-4efb-89ba-97cddc92628e","Type":"ContainerStarted","Data":"29ec44dd0a3c5c2b7577e410470209ddcc50790cd9ba47cc628a355d9e92ce8c"} Oct 29 22:43:02.984687 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.984633 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9eb7d909-e423-4efb-89ba-97cddc92628e","Type":"ContainerStarted","Data":"01ede87660c0e60ff1195c425e4212b24bcf3ab0224e7085761842e160f9433d"} Oct 29 22:43:02.989780 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.989504 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" event={"ID":"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5","Type":"ContainerStarted","Data":"539efc5e7ff05ea23303028836ca0ee0f8fd0a76cf1f2e72919bdaafe59b684d"} Oct 29 22:43:02.989780 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:02.989735 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:03.053331 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:03.053283 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.768479741 podStartE2EDuration="6.05326515s" totalImagesPullingTime="2.26974655s" totalInitContainerRuntime="15.038859ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:03.051759982 +0000 UTC m=+52.012626605" watchObservedRunningTime="2026-10-29 22:43:03.05326515 +0000 UTC m=+52.014131748" Oct 29 22:43:03.123549 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:03.123201 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" podStartSLOduration=2.619199147 podStartE2EDuration="5.123182961s" totalImagesPullingTime="2.503983814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:03.121953078 +0000 UTC m=+52.082819675" watchObservedRunningTime="2026-10-29 22:43:03.123182961 +0000 UTC m=+52.084049581" Oct 29 22:43:04.001848 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:04.001821 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" event={"ID":"bb967bfd-fb93-4051-bda5-9dbdf3589a69","Type":"ContainerStarted","Data":"7c39f4a47dd4b2bf36c38fa318925e013c4639165a20cd28e54ba3ebdab4f82c"} Oct 29 22:43:04.002190 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:04.002029 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:04.003736 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:04.003710 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" event={"ID":"1e4f37ae-5582-442e-bf04-a2cc29932ec6","Type":"ContainerStarted","Data":"5697782b890e4d5b4d933ea7df2e0595d527b647b94c2313b217bb9c737b2af3"} Oct 29 22:43:04.006889 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:04.006869 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:04.052486 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:04.052431 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" podStartSLOduration=2.465704089 podStartE2EDuration="4.052414317s" totalImagesPullingTime="1.586710228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:04.052119109 +0000 UTC m=+53.012985731" watchObservedRunningTime="2026-10-29 22:43:04.052414317 +0000 UTC m=+53.013280916" Oct 29 22:43:04.099919 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:04.099874 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" podStartSLOduration=2.529220098 podStartE2EDuration="4.099860345s" totalImagesPullingTime="1.570640247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:04.098732811 +0000 UTC m=+53.059599423" watchObservedRunningTime="2026-10-29 22:43:04.099860345 +0000 UTC m=+53.060726942" Oct 29 22:43:05.009786 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:05.009745 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" event={"ID":"9009d30d-e890-4365-83af-2244164260f3","Type":"ContainerStarted","Data":"c74bf54441ffe55498b330e1ddf8f51a06b0e84f54b2048d035c9aa7699ab9bf"} Oct 29 22:43:05.010266 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:05.009812 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" event={"ID":"9009d30d-e890-4365-83af-2244164260f3","Type":"ContainerStarted","Data":"7cf7fc8e30609fa7c9368d17dd8799c534ce93b6bdc7b02feffd7def837c6a89"} Oct 29 22:43:05.010266 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:05.009839 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" event={"ID":"9009d30d-e890-4365-83af-2244164260f3","Type":"ContainerStarted","Data":"6875accffd9ef551a42e23ac7c6876e162003948590851c8d00d00f5af150b2c"} Oct 29 22:43:05.094536 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:05.094485 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" podStartSLOduration=2.386683294 podStartE2EDuration="4.094471367s" totalImagesPullingTime="1.707788073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:05.09382805 +0000 UTC m=+54.054694647" watchObservedRunningTime="2026-10-29 22:43:05.094471367 +0000 UTC m=+54.055337964" Oct 29 22:43:05.855715 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:05.855635 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:07.930148 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:07.930119 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:09.012176 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:09.012150 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:17.348463 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.348427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:17.348463 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.348468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:43:17.379354 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.379331 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:43:17.379430 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.379410 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:43:17.391768 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.391746 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:17.391817 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.391799 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:43:17.449167 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.449141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:43:17.465517 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.465498 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:43:17.485672 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.485655 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:43:17.491902 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.491883 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:43:17.569347 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.569326 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mcv8v\"" Oct 29 22:43:17.573546 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.573529 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:43:17.579372 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.579350 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ltz9s\"" Oct 29 22:43:17.583715 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.583698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:17.587819 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.587800 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kfq4\"" Oct 29 22:43:17.591950 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.591922 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:43:17.878807 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.878738 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lglpm"] Oct 29 22:43:17.883596 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:43:17.883559 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4c20d8_f5e8_450b_b4f4_8199b277bbb0.slice/crio-662efec249f9348a8753fa5b58ff25b7640307baa96125e9e8237a0cf7ba5fb1 WatchSource:0}: Error finding container 662efec249f9348a8753fa5b58ff25b7640307baa96125e9e8237a0cf7ba5fb1: Status 404 returned error can't find the container with id 662efec249f9348a8753fa5b58ff25b7640307baa96125e9e8237a0cf7ba5fb1 Oct 29 22:43:17.892682 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.892656 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-twb2h"] Oct 29 22:43:17.894779 ip-10-0-143-172 kubenswrapper[2575]: W1029 22:43:17.894749 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb913afd_66ca_46e3_b9d1_66cff498245d.slice/crio-e84786c71912f055085170107984e21bbd7d259bb63cbc30f96f46a18473d800 WatchSource:0}: Error finding container e84786c71912f055085170107984e21bbd7d259bb63cbc30f96f46a18473d800: Status 404 returned error can't find the container with id e84786c71912f055085170107984e21bbd7d259bb63cbc30f96f46a18473d800 Oct 29 22:43:17.914817 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:17.914098 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d286w"] Oct 29 22:43:18.056968 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:18.056931 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lglpm" event={"ID":"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0","Type":"ContainerStarted","Data":"662efec249f9348a8753fa5b58ff25b7640307baa96125e9e8237a0cf7ba5fb1"} Oct 29 22:43:18.058058 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:18.058035 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-twb2h" event={"ID":"fb913afd-66ca-46e3-b9d1-66cff498245d","Type":"ContainerStarted","Data":"e84786c71912f055085170107984e21bbd7d259bb63cbc30f96f46a18473d800"} Oct 29 22:43:18.059054 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:18.059034 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d286w" event={"ID":"e85effca-b850-4af4-9086-ccfd965b7934","Type":"ContainerStarted","Data":"de8d654ac99108b85305574bde92242b39863430e96b449e71219e33ab3bd13c"} Oct 29 22:43:20.071869 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:20.071825 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d286w" event={"ID":"e85effca-b850-4af4-9086-ccfd965b7934","Type":"ContainerStarted","Data":"c12538bc3c4242b0ea0a05aa9be5507374b5c9628ae4a8d2e483270deeb9dbbd"} Oct 29 22:43:20.072296 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:20.071887 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d286w" event={"ID":"e85effca-b850-4af4-9086-ccfd965b7934","Type":"ContainerStarted","Data":"59f148dcc66b64ffe786fd3b819f7b9383b864e760dfd20320e5a39b8151e03a"} Oct 29 22:43:20.120855 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:20.120797 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d286w" podStartSLOduration=68.001388585 podStartE2EDuration="1m9.12077975s" totalImagesPullingTime="1.119391165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:20.119302638 +0000 UTC m=+69.080169232" watchObservedRunningTime="2026-10-29 22:43:20.12077975 +0000 UTC m=+69.081646346" Oct 29 22:43:21.078334 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:21.078277 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lglpm" event={"ID":"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0","Type":"ContainerStarted","Data":"228cd00715b4912f78f90ef3bde80d92a5404e4867b9e78f743b1792dd9eb05f"} Oct 29 22:43:21.078699 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:21.078548 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:43:21.123132 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:21.123080 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lglpm" podStartSLOduration=67.035423833 podStartE2EDuration="1m10.123065126s" totalImagesPullingTime="3.087641293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:21.12226652 +0000 UTC m=+70.083133118" watchObservedRunningTime="2026-10-29 22:43:21.123065126 +0000 UTC m=+70.083931720" Oct 29 22:43:21.140488 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:21.140463 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:21.140631 ip-10-0-143-172 kubenswrapper[2575]: I1029 22:43:21.140521 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:22.819625 ip-10-0-143-172 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 22:43:22.841933 ip-10-0-143-172 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 22:43:22.842099 ip-10-0-143-172 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 22:43:22.842272 ip-10-0-143-172 systemd[1]: kubelet.service: Consumed 4.435s CPU time. Oct 29 22:43:22.853291 ip-10-0-143-172 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:43:22.913165 ip-10-0-143-172 kubenswrapper[5736]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:43:22.913165 ip-10-0-143-172 kubenswrapper[5736]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:43:22.913165 ip-10-0-143-172 kubenswrapper[5736]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:43:22.913165 ip-10-0-143-172 kubenswrapper[5736]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:43:22.915212 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915135 5736 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:43:22.915212 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915211 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915217 5736 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915220 5736 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915223 5736 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915226 5736 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915229 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915232 5736 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915236 5736 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915241 5736 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915244 5736 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915247 5736 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915249 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915258 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915261 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915264 5736 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915267 5736 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915269 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915273 5736 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915275 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:43:22.915278 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915278 5736 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915281 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915284 5736 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915287 5736 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915290 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915293 5736 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915295 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915298 5736 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915301 5736 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915303 5736 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915306 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915309 5736 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915311 5736 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915314 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915317 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915320 5736 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915322 5736 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915325 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915327 5736 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:43:22.915767 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915330 5736 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915333 5736 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915335 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915338 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915341 5736 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915343 5736 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915346 5736 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915349 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915351 5736 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915355 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915357 5736 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915360 5736 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915364 5736 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915369 5736 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915372 5736 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915375 5736 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915378 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915381 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915383 5736 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:43:22.916233 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915386 5736 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915389 5736 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915392 5736 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915394 5736 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915397 5736 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915400 5736 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915403 5736 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915406 5736 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915408 5736 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915411 5736 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915413 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915416 5736 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915419 5736 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915421 5736 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915424 5736 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915426 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915429 5736 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915432 5736 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915434 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915437 5736 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:43:22.916773 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915440 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915443 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915446 5736 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915448 5736 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915451 5736 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915454 5736 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915456 5736 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915459 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915462 5736 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915465 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915468 5736 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915471 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915473 5736 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915476 5736 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915479 5736 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915482 5736 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915484 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915487 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915490 5736 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915492 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:43:22.917280 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915495 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915497 5736 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915500 5736 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915503 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915506 5736 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915510 5736 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915512 5736 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.915515 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916269 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916276 5736 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916279 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916282 5736 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916285 5736 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916290 5736 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916295 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916298 5736 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916301 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916304 5736 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916307 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916309 5736 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:43:22.917810 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916313 5736 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916316 5736 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916319 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916322 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916324 5736 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916327 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916330 5736 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916332 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916335 5736 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916338 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916340 5736 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916343 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916346 5736 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916348 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916351 5736 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916353 5736 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916356 5736 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916359 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916361 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:43:22.918325 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916364 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916367 5736 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916370 5736 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916372 5736 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916375 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916378 5736 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916381 5736 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916383 5736 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916386 5736 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916389 5736 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916392 5736 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916395 5736 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916398 5736 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916403 5736 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916407 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916410 5736 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916413 5736 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916416 5736 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916419 5736 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:43:22.918816 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916422 5736 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916425 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916428 5736 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916431 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916434 5736 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916436 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916439 5736 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916442 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916444 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916447 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916449 5736 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916452 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916455 5736 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916458 5736 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916460 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916463 5736 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916466 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916468 5736 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916471 5736 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916474 5736 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:43:22.919308 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916476 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916479 5736 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916482 5736 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916485 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916487 5736 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916490 5736 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916493 5736 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916496 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916498 5736 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916501 5736 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916503 5736 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916506 5736 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916508 5736 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916511 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916514 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916517 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916519 5736 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916522 5736 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916524 5736 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:43:22.919818 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916527 5736 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916530 5736 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916532 5736 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916535 5736 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916538 5736 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916541 5736 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916544 5736 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916547 5736 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916550 5736 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916553 5736 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916556 5736 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916558 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916561 5736 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916563 5736 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916566 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916569 5736 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.916572 5736 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917060 5736 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917066 5736 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:43:22.920318 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917069 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917071 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917075 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917078 5736 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917081 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917083 5736 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917086 5736 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917089 5736 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917092 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917094 5736 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917097 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917100 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917103 5736 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917106 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917109 5736 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917111 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917114 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917116 5736 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917119 5736 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917130 5736 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:43:22.920841 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917133 5736 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917136 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917139 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917142 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917144 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917147 5736 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917150 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917152 5736 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917155 5736 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917159 5736 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917164 5736 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917167 5736 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917169 5736 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917172 5736 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917175 5736 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917178 5736 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917180 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917183 5736 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917186 5736 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:43:22.921375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917189 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917192 5736 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917195 5736 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917197 5736 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917200 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917203 5736 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917205 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917208 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917211 5736 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917213 5736 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917216 5736 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917219 5736 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917221 5736 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917224 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917227 5736 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917231 5736 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917235 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917237 5736 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917240 5736 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:43:22.921869 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917243 5736 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917248 5736 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917251 5736 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917254 5736 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917258 5736 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917261 5736 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917265 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917267 5736 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917270 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917272 5736 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917275 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917278 5736 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917281 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917284 5736 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917286 5736 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917289 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917292 5736 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917295 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917298 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:43:22.922369 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917302 5736 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917306 5736 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917310 5736 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917313 5736 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917316 5736 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917319 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917321 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917324 5736 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917327 5736 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917329 5736 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917332 5736 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917335 5736 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917338 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917341 5736 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917346 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917349 5736 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917352 5736 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917356 5736 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917359 5736 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917362 5736 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917364 5736 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:43:22.922859 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917367 5736 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917370 5736 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917373 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917376 5736 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917379 5736 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.917382 5736 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917481 5736 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917491 5736 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917499 5736 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917504 5736 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917509 5736 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917512 5736 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917517 5736 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917521 5736 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917525 5736 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917528 5736 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917531 5736 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917535 5736 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917538 5736 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917541 5736 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917544 5736 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917547 5736 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917550 5736 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917553 5736 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:43:22.923375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917557 5736 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917562 5736 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917565 5736 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917568 5736 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917591 5736 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917595 5736 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917599 5736 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917603 5736 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917606 5736 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917609 5736 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917613 5736 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917616 5736 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917619 5736 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917622 5736 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917626 5736 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917629 5736 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917632 5736 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917635 5736 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917639 5736 flags.go:64] FLAG: --enable-server="true" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917642 5736 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917646 5736 flags.go:64] FLAG: --event-burst="100" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917649 5736 flags.go:64] FLAG: --event-qps="50" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917652 5736 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917655 5736 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:43:22.924144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917659 5736 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917663 5736 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917666 5736 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917669 5736 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917672 5736 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917675 5736 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917678 5736 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917681 5736 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917684 5736 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917689 5736 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917692 5736 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917695 5736 flags.go:64] FLAG: --feature-gates="" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917701 5736 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917704 5736 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917707 5736 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917711 5736 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917714 5736 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917717 5736 flags.go:64] FLAG: --help="false" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917720 5736 flags.go:64] FLAG: --hostname-override="ip-10-0-143-172.ec2.internal" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917723 5736 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917726 5736 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917729 5736 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917732 5736 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:43:22.925025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917736 5736 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917739 5736 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917742 5736 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917745 5736 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917748 5736 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917751 5736 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917754 5736 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917757 5736 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917760 5736 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917763 5736 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917766 5736 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917769 5736 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917771 5736 flags.go:64] FLAG: --lock-file="" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917774 5736 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917777 5736 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917781 5736 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917786 5736 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917789 5736 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917793 5736 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917797 5736 flags.go:64] FLAG: --logging-format="text" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917800 5736 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917804 5736 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917807 5736 flags.go:64] FLAG: --manifest-url="" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917811 5736 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917815 5736 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:43:22.925635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917819 5736 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917824 5736 flags.go:64] FLAG: --max-pods="110" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917827 5736 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917831 5736 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917834 5736 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917837 5736 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917840 5736 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917843 5736 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917846 5736 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917853 5736 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917856 5736 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917859 5736 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917862 5736 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917865 5736 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917868 5736 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917871 5736 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917874 5736 flags.go:64] FLAG: --port="10250" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917878 5736 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917881 5736 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-093c7071a13097967" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917884 5736 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917887 5736 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917890 5736 flags.go:64] FLAG: --register-node="true" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917893 5736 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917897 5736 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917900 5736 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917904 5736 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:43:22.926245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917907 5736 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917912 5736 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917916 5736 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917920 5736 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917922 5736 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917926 5736 flags.go:64] FLAG: --runonce="false" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917929 5736 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917933 5736 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917936 5736 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917939 5736 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917942 5736 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917945 5736 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917948 5736 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917951 5736 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917954 5736 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917957 5736 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917960 5736 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917963 5736 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917966 5736 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917969 5736 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917972 5736 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917977 5736 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917980 5736 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917983 5736 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917992 5736 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:43:22.926896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917995 5736 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.917998 5736 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.918001 5736 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.918004 5736 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.918007 5736 flags.go:64] FLAG: --v="2" Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.918011 5736 flags.go:64] FLAG: --version="false" Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.918017 5736 flags.go:64] FLAG: --vmodule="" Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.918021 5736 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.918024 5736 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918147 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918151 5736 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918154 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918157 5736 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918159 5736 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918162 5736 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918165 5736 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918168 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918171 5736 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918175 5736 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918179 5736 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918182 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:43:22.927496 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918185 5736 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918188 5736 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918191 5736 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918194 5736 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918197 5736 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918200 5736 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918203 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918206 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918208 5736 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918211 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918213 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918216 5736 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918219 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918222 5736 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918224 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918227 5736 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918229 5736 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918233 5736 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918236 5736 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:43:22.928139 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918239 5736 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918243 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918245 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918248 5736 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918251 5736 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918254 5736 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918256 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918259 5736 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918262 5736 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918265 5736 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918268 5736 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918270 5736 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918273 5736 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918276 5736 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918279 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918281 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918284 5736 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918287 5736 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918289 5736 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918292 5736 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:43:22.928678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918295 5736 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918297 5736 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918300 5736 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918302 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918305 5736 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918308 5736 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918310 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918313 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918316 5736 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918318 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918322 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918325 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918328 5736 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918332 5736 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918334 5736 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918337 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918340 5736 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918343 5736 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918345 5736 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918348 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:43:22.929252 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918350 5736 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918354 5736 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918356 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918359 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918362 5736 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918364 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918367 5736 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918370 5736 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918372 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918375 5736 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918378 5736 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918381 5736 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918384 5736 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918386 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918389 5736 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918392 5736 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918394 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918397 5736 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918400 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918403 5736 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:43:22.929831 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918405 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918408 5736 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918412 5736 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918414 5736 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918417 5736 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918421 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918424 5736 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918427 5736 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918429 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918432 5736 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918435 5736 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918437 5736 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918442 5736 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918446 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918449 5736 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.918475 5736 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:43:22.930388 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918512 5736 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918517 5736 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918520 5736 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918524 5736 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918528 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918530 5736 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918533 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918536 5736 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918538 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918541 5736 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918544 5736 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918546 5736 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918549 5736 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918552 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918554 5736 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918558 5736 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918562 5736 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918565 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918568 5736 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:43:22.930847 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918573 5736 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918589 5736 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918594 5736 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918597 5736 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918600 5736 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918603 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918606 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918608 5736 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918611 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918614 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918616 5736 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918619 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918623 5736 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918625 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918628 5736 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918631 5736 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918633 5736 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918636 5736 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918639 5736 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:43:22.931379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918642 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918644 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918647 5736 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918650 5736 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918652 5736 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918655 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918658 5736 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918660 5736 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918663 5736 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918666 5736 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918668 5736 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918671 5736 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918674 5736 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918678 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918681 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918685 5736 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918688 5736 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918691 5736 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918693 5736 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918696 5736 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:43:22.932054 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918698 5736 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918701 5736 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918704 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918706 5736 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918709 5736 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918712 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918715 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918718 5736 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918720 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918723 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918726 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918729 5736 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918732 5736 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918735 5736 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918737 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918740 5736 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918743 5736 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918745 5736 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918748 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918751 5736 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:43:22.932632 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918753 5736 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918756 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918759 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918761 5736 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918764 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918768 5736 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918770 5736 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918775 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918777 5736 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918780 5736 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918783 5736 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918785 5736 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918788 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918791 5736 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918794 5736 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918797 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918799 5736 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918802 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918805 5736 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918808 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:43:22.933115 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918810 5736 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918813 5736 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918816 5736 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918819 5736 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918822 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918824 5736 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918827 5736 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.918831 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.922788 5736 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.922803 5736 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.922820 5736 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.922825 5736 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922890 5736 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922896 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922900 5736 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922903 5736 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922906 5736 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922909 5736 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922912 5736 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:43:22.933626 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922914 5736 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922917 5736 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922920 5736 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922923 5736 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922926 5736 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922928 5736 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922931 5736 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922934 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922937 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922940 5736 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922942 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922945 5736 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922953 5736 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922956 5736 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922959 5736 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922962 5736 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922965 5736 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922968 5736 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922973 5736 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922977 5736 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:43:22.934136 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922980 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922983 5736 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922986 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922989 5736 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922992 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922995 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.922998 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923001 5736 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923004 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923006 5736 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923009 5736 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923012 5736 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923014 5736 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923017 5736 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923020 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923022 5736 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923025 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923028 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923030 5736 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:43:22.934644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923033 5736 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923036 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923038 5736 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923041 5736 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923044 5736 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923047 5736 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923055 5736 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923058 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923061 5736 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923064 5736 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923067 5736 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923069 5736 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923072 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923075 5736 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923077 5736 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923080 5736 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923083 5736 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923085 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923088 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923091 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:43:22.935125 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923093 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923096 5736 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923099 5736 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923101 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923104 5736 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923106 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923109 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923112 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923114 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923117 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923120 5736 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923123 5736 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923127 5736 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923130 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923133 5736 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923136 5736 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923139 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923142 5736 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923145 5736 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:43:22.935644 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923153 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923156 5736 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923159 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923162 5736 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923164 5736 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923167 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923170 5736 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923172 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923175 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923178 5736 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923180 5736 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923183 5736 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923185 5736 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923189 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923191 5736 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923194 5736 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923196 5736 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923199 5736 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923201 5736 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923204 5736 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:43:22.936106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923206 5736 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.923234 5736 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923443 5736 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923448 5736 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923451 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923454 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923457 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923460 5736 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923463 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923465 5736 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923468 5736 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923470 5736 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923474 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923482 5736 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923485 5736 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923488 5736 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:43:22.936598 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923490 5736 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923493 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923496 5736 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923498 5736 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923501 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923504 5736 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923508 5736 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923512 5736 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923515 5736 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923518 5736 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923521 5736 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923524 5736 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923526 5736 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923529 5736 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923532 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923534 5736 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923537 5736 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923540 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923542 5736 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:43:22.936991 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923545 5736 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923548 5736 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923551 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923554 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923556 5736 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923559 5736 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923561 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923564 5736 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923567 5736 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923569 5736 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923572 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923603 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923607 5736 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923609 5736 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923612 5736 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923615 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923617 5736 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923620 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923623 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923626 5736 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:43:22.937464 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923628 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923631 5736 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923634 5736 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923637 5736 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923639 5736 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923642 5736 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923645 5736 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923647 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923650 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923653 5736 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923655 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923659 5736 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923663 5736 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923665 5736 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923669 5736 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923672 5736 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923675 5736 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923677 5736 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923680 5736 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923683 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:43:22.937943 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923685 5736 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923688 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923690 5736 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923693 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923732 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923943 5736 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923949 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923953 5736 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923960 5736 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923964 5736 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923969 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923976 5736 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923980 5736 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923985 5736 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923989 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.923994 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924005 5736 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924009 5736 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924014 5736 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:43:22.938421 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924018 5736 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924023 5736 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924027 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924031 5736 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924036 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924040 5736 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924045 5736 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924049 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924054 5736 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924058 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924067 5736 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924072 5736 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924077 5736 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924081 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.924141 5736 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924300 5736 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:43:22.938882 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924312 5736 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924319 5736 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924349 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924353 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924358 5736 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924362 5736 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924366 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924370 5736 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924375 5736 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924379 5736 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924384 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924388 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924397 5736 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924402 5736 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924406 5736 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924410 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924414 5736 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924418 5736 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924422 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924427 5736 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:43:22.939336 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924431 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924436 5736 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924440 5736 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924445 5736 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924449 5736 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924459 5736 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924463 5736 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924467 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924472 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924476 5736 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924480 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924487 5736 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924496 5736 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924502 5736 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924506 5736 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924537 5736 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924542 5736 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924551 5736 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924556 5736 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924560 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:43:22.939979 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924564 5736 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924568 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924572 5736 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924600 5736 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924605 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924610 5736 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924614 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924618 5736 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924622 5736 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924626 5736 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924635 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924640 5736 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924644 5736 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924650 5736 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924655 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924659 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924663 5736 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924668 5736 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924672 5736 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:43:22.940727 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924676 5736 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924683 5736 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924687 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924696 5736 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924700 5736 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924706 5736 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924711 5736 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924715 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924720 5736 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924754 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924759 5736 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924763 5736 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924768 5736 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924772 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924777 5736 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924786 5736 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.924790 5736 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925225 5736 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925343 5736 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925348 5736 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:43:22.941403 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925352 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925355 5736 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925358 5736 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925361 5736 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925364 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925367 5736 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925370 5736 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925373 5736 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925376 5736 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925379 5736 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925381 5736 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925384 5736 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925389 5736 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925392 5736 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925398 5736 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925403 5736 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925406 5736 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925409 5736 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925415 5736 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:43:22.942113 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925418 5736 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925421 5736 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925423 5736 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925427 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925431 5736 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925434 5736 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:43:22.925437 5736 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.925509 5736 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.926459 5736 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.926505 5736 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.926696 5736 server.go:1014] "Starting client certificate rotation" Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.926827 5736 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.926881 5736 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:37:11 +0000 UTC" deadline="2028-06-10 00:12:27.517785885 +0000 UTC" Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.926914 5736 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14137h29m4.590873823s" Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.927137 5736 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.928187 5736 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:43:22.942741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.929572 5736 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:43:22.943182 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.931438 5736 log.go:25] "Validated CRI v1 image API" Oct 29 22:43:22.943182 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.932113 5736 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:43:22.943182 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.934335 5736 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d877c562-a4a7-4bf9-8837-e2523146b187:/dev/nvme0n1p3 dc5250ab-f587-4dfa-b14a-60b9f1e90bc6:/dev/nvme0n1p4] Oct 29 22:43:22.943347 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.934354 5736 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01a49e9fa13dadb93adfb64727b600a7c5da3af980482e09cc09f40155e3186a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01a49e9fa13dadb93adfb64727b600a7c5da3af980482e09cc09f40155e3186a/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b7f9f9487f36b130419d5e13f19527462b0294d67d287abb2cd625b121036ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b7f9f9487f36b130419d5e13f19527462b0294d67d287abb2cd625b121036ef/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/465e1464aa13f2b03e5119bd5dd6ad2cc127bc7a0a8b68d14ef7e01e3d5ebe0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/465e1464aa13f2b03e5119bd5dd6ad2cc127bc7a0a8b68d14ef7e01e3d5ebe0a/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49c4e62507582f24fda9808b6efcb0eb12d9cb9dedce54a2bc74930b1d423374/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49c4e62507582f24fda9808b6efcb0eb12d9cb9dedce54a2bc74930b1d423374/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5847add22193d273cbe18ef95c4be2741a72265fe0db69c237b7a86b5ac15b51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5847add22193d273cbe18ef95c4be2741a72265fe0db69c237b7a86b5ac15b51/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/662efec249f9348a8753fa5b58ff25b7640307baa96125e9e8237a0cf7ba5fb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/662efec249f9348a8753fa5b58ff25b7640307baa96125e9e8237a0cf7ba5fb1/userdata/shm major:0 minor:454 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6df744fbb0e4f3f560f8df3d294b089e974d659b20d2f3ae7e3d812ed6ae4e74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6df744fbb0e4f3f560f8df3d294b089e974d659b20d2f3ae7e3d812ed6ae4e74/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85b7a4c9fffe7efecd43ab2ccecb1c9b8ee8b0432a9c97d0246b279322296d09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85b7a4c9fffe7efecd43ab2ccecb1c9b8ee8b0432a9c97d0246b279322296d09/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bd8cb41078d1d26e13751de8b68473b6d736c7f6cc741d982f6bb0cdf7a9b5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bd8cb41078d1d26e13751de8b68473b6d736c7f6cc741d982f6bb0cdf7a9b5d/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/912b8930f9f8f8e907be377a9381118ceccef6fe57cafc8ab865b5a67e2a3952/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/912b8930f9f8f8e907be377a9381118ceccef6fe57cafc8ab865b5a67e2a3952/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ed974afc27d40236bc9889521d4ec285ca2b3450459909f787fa680be34fb41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ed974afc27d40236bc9889521d4ec285ca2b3450459909f787fa680be34fb41/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac8ec5ee4af5ab87210981e33ad8a03de9364f5ddc8cdbb5041a56d6a9e75b27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac8ec5ee4af5ab87210981e33ad8a03de9364f5ddc8cdbb5041a56d6a9e75b27/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bafb3bf21c841d6aa1a98ea0c2751e1932d384abf57dd6c6ea8f2a6f400b4d3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bafb3bf21c841d6aa1a98ea0c2751e1932d384abf57dd6c6ea8f2a6f400b4d3e/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb9f61588c747ca41785a29fa90332c8070870d825bc1ccca3e5ebba52f58702/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb9f61588c747ca41785a29fa90332c8070870d825bc1ccca3e5ebba52f58702/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdac24c91b8bfb9688187aaef68eba4f4e4c0bc3452bf60aa6687219885cf893/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdac24c91b8bfb9688187aaef68eba4f4e4c0bc3452bf60aa6687219885cf893/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4fdf7c93e52fcc9844fa0399c8c18867ceed621cad347eac4551a8820553feb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4fdf7c93e52fcc9844fa0399c8c18867ceed621cad347eac4551a8820553feb/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de8d654ac99108b85305574bde92242b39863430e96b449e71219e33ab3bd13c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de8d654ac99108b85305574bde92242b39863430e96b449e71219e33ab3bd13c/userdata/shm major:0 minor:458 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3eb10ad2da597c1507fa4439f14ebc4837b39c346ee8ab6a1b1b48580b88395/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3eb10ad2da597c1507fa4439f14ebc4837b39c346ee8ab6a1b1b48580b88395/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6d7fa437c70919c1cce9fcd1945c52472980b7628e5ce4e09cebd5c9d5cecfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6d7fa437c70919c1cce9fcd1945c52472980b7628e5ce4e09cebd5c9d5cecfa/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e84786c71912f055085170107984e21bbd7d259bb63cbc30f96f46a18473d800/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e84786c71912f055085170107984e21bbd7d259bb63cbc30f96f46a18473d800/userdata/shm major:0 minor:456 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb075e522fd531388bf83d138dc25235a9f448e3bf61996c12bf5ece6b62901c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb075e522fd531388bf83d138dc25235a9f448e3bf61996c12bf5ece6b62901c/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f092da451c9d7ac853e0529c3e2ed53ec04ddfc1e9918c52c86c293a711b6faf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f092da451c9d7ac853e0529c3e2ed53ec04ddfc1e9918c52c86c293a711b6faf/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4c14097f29a0b32410b03debe8ffa4f099aa6ee1d7b888c4c32ac2ba978f410/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4c14097f29a0b32410b03debe8ffa4f099aa6ee1d7b888c4c32ac2ba978f410/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7a8fd2af6af21a4139d5b7c6d92fd2224131579fcc42e1784ae607e72b93edc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7a8fd2af6af21a4139d5b7c6d92fd2224131579fcc42e1784ae607e72b93edc/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~projected/kube-api-access-qqbwl:{mountpoint:/var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~projected/kube-api-access-qqbwl major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fd11cc5-35ab-4347-8914-e953b71bc322/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4fd11cc5-35ab-4347-8914-e953b71bc322/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54a953c6-9194-404e-b8af-7ef3540d0e6d/volumes/kubernetes.io~projected/kube-api-access-5x79c:{mountpoint:/var/lib/kubelet/pods/54a953c6-9194-404e-b8af-7ef3540d0e6d/volumes/kubernetes.io~projected/kube-api-access-5x79c major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a6a57a5-af08-4ceb-a886-c176359b8d5a/volumes/kubernetes.io~projected/kube-api-access-lr776:{mountpoint:/var/lib/kubelet/pods/6a6a57a5-af08-4ceb-a886-c176359b8d5a/volumes/kubernetes.io~projected/kube-api-access-lr776 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0/volumes/kubernetes.io~projected/kube-api-access-2skn4:{mountpoint:/var/lib/kubelet/pods/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0/volumes/kubernetes.io~projected/kube-api-access-2skn4 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/875f6c48-91a9-4f0b-a71d-e41d02e589d5/volumes/kubernetes.io~projected/kube-api-access-p4kmz:{mountpoint:/var/lib/kubelet/pods/875f6c48-91a9-4f0b-a71d-e41d02e589d5/volumes/kubernetes.io~projected/kube-api-access-p4kmz major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/875f6c48-91a9-4f0b-a71d-e41d02e589d5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/875f6c48-91a9-4f0b-a71d-e41d02e589d5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/875f6c48-91a9-4f0b-a71d-e41d02e589d5/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/875f6c48-91a9-4f0b-a71d-e41d02e589d5/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/878dc633-6f09-450f-ab79-8788f14e3055/volumes/kubernetes.io~projected/kube-api-access-s2sm9:{mountpoint:/var/lib/kubelet/pods/878dc633-6f09-450f-ab79-8788f14e3055/volumes/kubernetes.io~projected/kube-api-access-s2sm9 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b726c8b-651c-4d5e-afa5-26a9f159d609/volumes/kubernetes.io~projected/kube-api-access-vpt7k:{mountpoint:/var/lib/kubelet/pods/8b726c8b-651c-4d5e-afa5-26a9f159d609/volumes/kubernetes.io~projected/kube-api-access-vpt7k major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cdf7ea9-aed7-4071-ada6-2f0518de3a87/volumes/kubernetes.io~projected/kube-api-access-rj97r:{mountpoint:/var/lib/kubelet/pods/8cdf7ea9-aed7-4071-ada6-2f0518de3a87/volumes/kubernetes.io~projected/kube-api-access-rj97r major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cdf7ea9-aed7-4071-ada6-2f0518de3a87/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8cdf7ea9-aed7-4071-ada6-2f0518de3a87/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cdf7ea9-aed7-4071-ada6-2f0518de3a87/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/8cdf7ea9-aed7-4071-ada6-2f0518de3a87/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~projected/kube-api-access-wcqw6:{mountpoint:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~projected/kube-api-access-wcqw6 major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/web-config/alertmanager/9 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/web-config/config-reloader/6 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~empty-dir/config-out major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~projected/kube-api-access-fngpp:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~projected/kube-api-access-fngpp major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~projected/tls-assets major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/config-volume major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/web-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a66d8310-cc78-4a48-bc21-03fa742d0829/volumes/kubernetes.io~projected/kube-api-access-6kkhw:{mountpoint:/var/lib/kubelet/pods/a66d8310-cc78-4a48-bc21-03fa742d0829/volumes/kubernetes.io~projected/kube-api-access-6kkhw major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b08b826d-232a-48be-ade2-dd833d8c3423/volumes/kubernetes.io~projected/kube-api-access-rrlvp:{mountpoint:/var/lib/kubelet/pods/b08b826d-232a-48be-ade2-dd833d8c3423/volumes/kubernetes.io~projected/kube-api-access-rrlvp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b08b826d-232a-48be-ade2-dd833d8c3423/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b08b826d-232a-48be-ade2-dd833d8c3423/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1d25bfd-dba6-4268-8123-533f4b106c0a/volumes/kubernetes.io~projected/kube-api-access-w58hz:{mountpoint:/var/lib/kubelet/pods/b1d25bfd-dba6-4268-8123-533f4b106c0a/volumes/kubernetes.io~projected/kube-api-access-w58hz major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1d25bfd-dba6-4268-8123-533f4b106c0a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b1d25bfd-dba6-4268-8123-533f4b106c0a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1d25bfd-dba6-4268-8123-533f4b106c0a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b1d25bfd-dba6-4268-8123-533f4b106c0a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b78e901b-2f4b-4ee5-a96d-573988c5cadd/volumes/kubernetes.io~projected/kube-api-access-rx6bq:{mountpoint:/var/lib/kubelet/pods/b78e901b-2f4b-4ee5-a96d-573988c5cadd/volumes/kubernetes.io~projected/kube-api-access-rx6bq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb967bfd-fb93-4051-bda5-9dbdf3589a69/volumes/kubernetes.io~projected/kube-api-access-w2gfd:{mountpoint:/var/lib/kubelet/pods/bb967bfd-fb93-4051-bda5-9dbdf3589a69/volumes/kubernetes.io~projected/kube-api-access-w2gfd major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb967bfd-fb93-4051-bda5-9dbdf3589a69/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/bb967bfd-fb93-4051-bda5-9dbdf3589a69/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~projected/kube-api-access-dfnkq:{mountpoint:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~projected/kube-api-access-dfnkq major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cee899d2-5ec5-443d-b004-89ce10507f7d/volumes/kubernetes.io~projected/kube-api-access-g8xcj:{mountpoint:/var/lib/kubelet/pods/cee899d2-5ec5-443d-b004-89ce10507f7d/volumes/kubernetes.io~projected/kube-api-access-g8xcj major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cee899d2-5ec5-443d-b004-89ce10507f7d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/cee899d2-5ec5-443d-b004-89ce10507f7d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d31f1eee-73b6-4b6d-bde0-1006646b0173/volumes/kubernetes.io~projected/kube-api-access-9qt87:{mountpoint:/var/lib/kubelet/pods/d31f1eee-73b6-4b6d-bde0-1006646b0173/volumes/kubernetes.io~projected/kube-api-access-9qt87 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d31f1eee-73b6-4b6d-bde0-1006646b0173/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d31f1eee-73b6-4b6d-bde0-1006646b0173/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d839fc45-f2f4-4ce4-95d8-9e3145342973/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d839fc45-f2f4-4ce4-95d8-9e3145342973/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d839fc45-f2f4-4ce4-95d8-9e3145342973/volumes/kubernetes.io~projected/kube-api-access-rtq46:{mountpoint:/var/lib/kubelet/pods/d839fc45-f2f4-4ce4-95d8-9e3145342973/volumes/kubernetes.io~projected/kube-api-access-rtq46 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d839fc45-f2f4-4ce4-95d8-9e3145342973/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d839fc45-f2f4-4ce4-95d8-9e3145342973/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e85effca-b850-4af4-9086-ccfd965b7934/volumes/kubernetes.io~projected/kube-api-access-jwfgr:{mountpoint:/var/lib/kubelet/pods/e85effca-b850-4af4-9086-ccfd965b7934/volumes/kubernetes.io~projected/kube-api-access-jwfgr major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e85effca-b850-4af4-9086-ccfd965b7934/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/e85effca-b850-4af4-9086-ccfd965b7934/volumes/kubernetes.io~secret/metrics-certs major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0087fb0-f6d1-4f82-b78b-7f460f997ef0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f0087fb0-f6d1-4f82-b78b-7f460f997ef0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0087fb0-f6d1-4f82-b78b-7f460f997ef0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f0087fb0-f6d1-4f82-b78b-7f460f997ef0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0087fb0-f6d1-4f82-b78b-7f460f997ef0/volumes/kubernetes.io~projected/kube-api-access-95k69:{mountpoint:/var/lib/kubelet/pods/f0087fb0-f6d1-4f82-b78b-7f460f997ef0/volumes/kubernetes.io~projected/kube-api-access-95k69 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb913afd-66ca-46e3-b9d1-66cff498245d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fb913afd-66ca-46e3-b9d1-66cff498245d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:448 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2de6a72d842ed26280f7fdb9c5fccf913a3fae2a11bfbd1c003a3945ec6720e2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7d91cf18e8c387a4242bd145c078f84c9c1db08f7a28f27602df213fefe4b9a0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/104f920c69a8999e5aeea68eaa138e50f595c41892d5f7c6b14f86db38cf80fe/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/18f6a3cbcdf8f8f59143896d9c81b6631536ac31fa33798bd9a50c7d762e81e6/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9af643004740052f19d0b8fb855e7893a398698c4b4a3c08304352008e258d9c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0118802dfa9c61a0847db5d65394684cb0767314e7b8cd81f20b093427d4ed81/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b96f3a2142ab0ac090377941e3a6d6605a4a1c05a612f712190304176084a480/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f33ca7ed6eaa25cb787c98e27a4c3c4330d1100f2968588b3f7948481c06d342/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/705d82b6429ec47edba60e8d8be615f2bba60fb4027dab2fffc52c0d596590dd/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ced39fc2c9ccbf6ab29ef6f6102c90d8bdd60c6464e299c777e12f7863c03e4a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/6a8a38e29f2b7d237739ecc57389029fcd89aafd94c968bff84c21aff0ec62cf/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/0a00072f1962c24a28d2223f998870aac956b08c6474eba47a28505b11c4459c/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/38318a767aca1cabe2252fc990059cf1738d523a0d61cba0bf9d5ca6f6b76439/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d3132e0db1e1af70cafb562c31c074cdc7d62c63ea66cc40669068242c1c6d8d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/172bf4525c777663ded1a24feb46fb8bd98391016c80d4f35d526e885c3a69ed/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e9e34ac634c2feeba3363624546111cfc5445bfa65dd0c28f5fd528c4b0e7502/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/792fa42f63aa688662c850cd4c1933ee6574d46a8065140dfa49eeeae5e7f167/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5fb677055c6589bb756d38a42fa0b2277154d8abb3cbf0da6308c4060c4ced7b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3ab154ab32eb4b2569d6beef18bae255205fefb0b8cbcafd0243b93ba3388c74/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/314ed13ba5177703a67bd4ef0adcbd5f95a9303f65fa917c3617d3ea5946e1a7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a45ce60a8817b91e12889f34131801d68f8228ee5441c5d1df1baace42469e8c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/a181fbf26fece09cbacf5100ac4ad1f3c7583c999bc1edaae1c50f131f4ea600/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/cf6a0896a79ccc9924c6fcaa1f5136041e126e03e409722a14e8cdcf80ee38f8/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/946496a507325836822d1fff79a0682138aad5d13a9a4ba7aa00c8b411077032/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/801ca5ed4e4158e0eb0c68b766daf590b891f52e4796409e48a038d62cce0a84/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b345070978920f888ecbba64a063a39d98a66d16b970047fac0a4bbd011742d6/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/8ecbb0ffedc9f48f2d1ed5f06348f1b4e2c37fd91af94e67038767b443ff261b/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/6f40b7408465bb5d23fb0b43b6949c19116461e59045b9b0440c4ce1b5854968/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/6b0ece53609dbdcb33eee1e9cfa4d824ce1af0484e8ba38c28fe48e19f3d389f/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/64da0af5867514b30757a3b197e9855f791ec3f37a6ef4e43ea79132afdb6d30/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/36b7f608841387f34c63849158b5dfbb8692c1bb13cc7c4cf8d12b58cc68c526/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/18caee798096a252671a95b6ee07d5c2f6bb6d9c9853d928622f662f951c818e/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/63993bb9d5dc762c05ec1606e1196f37306fd4d6eaf2df54375248e0ed9b1acb/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/3b193ec6503caea72372edb6ece9d9a2b7e4dbe336c98560776ad4c7de4371bd/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/87596232e1dd5b3ad74d38f3c08dec4b78050c5d9e96fd2acb9f852c7b9704cb/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/d50f2990307c2df545490184ac80ff220d592cdefab9c12789b93bf326bd7793/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/7297546e0ff6a544d29cce6ea8d982b46f3c1582bfdd794c6fdf96e53b30f39e/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/38c5b8e9feecadcb30986fdc47a8c82521667f843331f3dd6c4ddc229b9d2299/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/cccafbcc85f92589d24af6306503b84bbba6d453510d6bac3d380423b854d677/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/9fd51838eb347dee049c8a284d7bdf7b4a90c5902316951bbf319e951c2cf737/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/5f51549e8d0fdac181143a8711a29a43b82345ea50e772c1359ea8bff9441543/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/d7e7873e043ca70c4fbcc987b784e645d2ef2fdd636c85f940ac2b24c26163a7/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/d303d88625bf1e079586fcb432b3be8128e7fa12e5bb87bcef4c399bd9849946/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/ce607383f6af39ef86bf332627adf12be5d30d79425880f586b5b1e86bb23de5/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/0255017643cbcba105392ae738026d1415a8afcdfc150443fef2e8517cce66a4/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/6fe3a9e7833a946296883f83ed878264d9b229f088bdd5f3fa2284c4be58d812/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/d22cd4c2b3349759926aacd89b3f9749f0c356837cf560c2c05b2901e88511a7/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/eed07f506d274838450fa2cbe6299e1d29916c15c17d8cd1be0b7eba192a4e16/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/6a5a123efb53f2ac07bdf78f0121a0abbd54c87be33e43950b904031d57187c6/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/fdc4b314d6603a2432d99eddb37b0ccd88f9a95425c8fb595e6f11f122abd983/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/f998009edfcc2dcbc9ee7340001222d771f9dfac9350574d202091d25a96e03e/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/97c38c3951abbc5d8e5a9e14af55830422c7c7011799e3509aeba1d2fbc4a1ee/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/7031eddd8d81c7cbc5c10cb3067eafc3ce8e37277d846517fc91cd9b63df5d73/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/b49367974b7ed1cc0a8cd077b447cf2188ff3e26e6f9cf863e3109d696481c0a/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/b7d6e6b7f8f00406024e3b4ae78e704127514756f54f33730d94110ee5ee3f78/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/8518173d5334a028b8ef7800e2d5a4cb21729a5cda862251495524c0bc2c3bbc/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/e0d25365e9a1db9f4cff7c91f9a81136efa32e5de195e2edb19389c49c0e0e04/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/a7fb1263b7299901b3e5762d1083351e17f9e08a470c9ae1f46ded9244b8ab63/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/1d8ca523f7cf6309c08b755d065f1219345caf611387dfea7b60cf83602f97ed/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/fef6064b9727880c0e24b800d50d91b6f7bfd89823a968dc7ae2ad4177bc3bb8/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/cf0c33f6877cf2ac9596845277825f64d2b86eeff1aef440d6dd9a565398441f/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/d0dc11ed5daa50fc6f3897d412c7ea6d96a930c0ef9b0cebf1bb623daf971991/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/47518754a234f8976502982b0529392900b73778b18bca25dcd65f16ca3ee15a/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/f33de32b1a37c0503ba6b063655fb2371eedf5a38909c947b33697bcc710dcf0/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/73838089ed2b782930bb901bee8e4c54a36ef81f18a7393a1e7f29a7465a6ba2/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/9394a8689a568cbc0ee12d43f1f329518441278fb4a53492dac3eee78619793f/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/b8f10527a92f684eda36b9da090e580ff7f33e6a137e4035a095d2ae48f35c7b/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/b9998a08b14628db6684231c4f5ffc986a7f56834be624e995c4abd0aeadf7ed/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/12493dd4eb1dd85f9a857d0d84e09280b30637cf130a25796edb95cbe2ebb0ca/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-479:{mountpoint:/var/lib/containers/storage/overlay/4664034717f3db0bc6f8aa6a3518de9e1ad1ed80aa9f50cddbe1b7a524be6ca0/merged major:0 minor:479 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b072f9dc2eb8b87c2ef80437680c8f0b819067d09c8fd3bae5daa3880269ed4d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6eb084952e70d1af973291f399f04b50a2950bcf37955d08b98bc0537a62146b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/08c906ad3b703e24b0b54b8240ac6f61f0c3e6d527d5b00845f0799aee91f84d/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/14d2d21d6b80cc026e0e55e36bc8fea779829935be7d596f71ae669c675ac352/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/cc8a2dc6bc9f6d2b970cd57412c3c5f4edf8d52006e55810e614c98bedd1b9f9/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/14e7004aa6c53f5875b57535f9eea1e2fd5794b8015e4347fe700064e68be086/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/58cbe70187cb70e0c5779ab66da6ec620aade39c6d2c14708afc80db3be50629/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/49eb53aa3882c13ab3cd3d6acf7b56b7020f149d581ab26c1c6270ff506812ad/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b363e1222a2c0aac640e8e66d9abf03b017a5fa86bed04351896f17a02bb3443/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/2032687f6581b15ea3a69afacffcb9385a573c571c0aa43c33144f73b737027a/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 29 22:43:22.944697 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.943572 5736 manager.go:217] Machine: {Timestamp:2026-10-29 22:43:22.942638716 +0000 UTC m=+0.058558312 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3027762 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25bde06274256c634eeeb1d4acc638 SystemUUID:ec25bde0-6274-256c-634e-eeb1d4acc638 BootID:6c5608ce-3c4b-4bec-8b04-13edfa6e5e2b Filesystems:[{Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d31f1eee-73b6-4b6d-bde0-1006646b0173/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7a8fd2af6af21a4139d5b7c6d92fd2224131579fcc42e1784ae607e72b93edc/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0087fb0-f6d1-4f82-b78b-7f460f997ef0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bd8cb41078d1d26e13751de8b68473b6d736c7f6cc741d982f6bb0cdf7a9b5d/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b08b826d-232a-48be-ade2-dd833d8c3423/volumes/kubernetes.io~projected/kube-api-access-rrlvp DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1d25bfd-dba6-4268-8123-533f4b106c0a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4c14097f29a0b32410b03debe8ffa4f099aa6ee1d7b888c4c32ac2ba978f410/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb9f61588c747ca41785a29fa90332c8070870d825bc1ccca3e5ebba52f58702/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:368 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~projected/kube-api-access-wcqw6 DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bafb3bf21c841d6aa1a98ea0c2751e1932d384abf57dd6c6ea8f2a6f400b4d3e/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b1d25bfd-dba6-4268-8123-533f4b106c0a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/54a953c6-9194-404e-b8af-7ef3540d0e6d/volumes/kubernetes.io~projected/kube-api-access-5x79c DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d839fc45-f2f4-4ce4-95d8-9e3145342973/volumes/kubernetes.io~projected/kube-api-access-rtq46 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6df744fbb0e4f3f560f8df3d294b089e974d659b20d2f3ae7e3d812ed6ae4e74/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b08b826d-232a-48be-ade2-dd833d8c3423/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/875f6c48-91a9-4f0b-a71d-e41d02e589d5/volumes/kubernetes.io~projected/kube-api-access-p4kmz DeviceMajor:0 DeviceMinor:244 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-479 DeviceMajor:0 DeviceMinor:479 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b7f9f9487f36b130419d5e13f19527462b0294d67d287abb2cd625b121036ef/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:325 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0087fb0-f6d1-4f82-b78b-7f460f997ef0/volumes/kubernetes.io~projected/kube-api-access-95k69 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a6a57a5-af08-4ceb-a886-c176359b8d5a/volumes/kubernetes.io~projected/kube-api-access-lr776 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cee899d2-5ec5-443d-b004-89ce10507f7d/volumes/kubernetes.io~projected/kube-api-access-g8xcj DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e85effca-b850-4af4-9086-ccfd965b7934/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:452 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/912b8930f9f8f8e907be377a9381118ceccef6fe57cafc8ab865b5a67e2a3952/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fd11cc5-35ab-4347-8914-e953b71bc322/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1d25bfd-dba6-4268-8123-533f4b106c0a/volumes/kubernetes.io~projected/kube-api-access-w58hz DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b78e901b-2f4b-4ee5-a96d-573988c5cadd/volumes/kubernetes.io~projected/kube-api-access-rx6bq DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cdf7ea9-aed7-4071-ada6-2f0518de3a87/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a66d8310-cc78-4a48-bc21-03fa742d0829/volumes/kubernetes.io~projected/kube-api-access-6kkhw DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac8ec5ee4af5ab87210981e33ad8a03de9364f5ddc8cdbb5041a56d6a9e75b27/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~projected/kube-api-access-fngpp DeviceMajor:0 DeviceMinor:292 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de8d654ac99108b85305574bde92242b39863430e96b449e71219e33ab3bd13c/userdata/shm DeviceMajor:0 DeviceMinor:458 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8cdf7ea9-aed7-4071-ada6-2f0518de3a87/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:351 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d839fc45-f2f4-4ce4-95d8-9e3145342973/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:273 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3eb10ad2da597c1507fa4439f14ebc4837b39c346ee8ab6a1b1b48580b88395/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bb967bfd-fb93-4051-bda5-9dbdf3589a69/volumes/kubernetes.io~projected/kube-api-access-w2gfd DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49c4e62507582f24fda9808b6efcb0eb12d9cb9dedce54a2bc74930b1d423374/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0087fb0-f6d1-4f82-b78b-7f460f997ef0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdac24c91b8bfb9688187aaef68eba4f4e4c0bc3452bf60aa6687219885cf893/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cdf7ea9-aed7-4071-ada6-2f0518de3a87/volumes/kubernetes.io~projected/kube-api-access-rj97r DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d31f1eee-73b6-4b6d-bde0-1006646b0173/volumes/kubernetes.io~projected/kube-api-access-9qt87 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~projected/kube-api-access-dfnkq DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4fdf7c93e52fcc9844fa0399c8c18867ceed621cad347eac4551a8820553feb/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb075e522fd531388bf83d138dc25235a9f448e3bf61996c12bf5ece6b62901c/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb913afd-66ca-46e3-b9d1-66cff498245d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:448 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cee899d2-5ec5-443d-b004-89ce10507f7d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6d7fa437c70919c1cce9fcd1945c52472980b7628e5ce4e09cebd5c9d5cecfa/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:320 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/662efec249f9348a8753fa5b58ff25b7640307baa96125e9e8237a0cf7ba5fb1/userdata/shm DeviceMajor:0 DeviceMinor:454 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8b726c8b-651c-4d5e-afa5-26a9f159d609/volumes/kubernetes.io~projected/kube-api-access-vpt7k DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/875f6c48-91a9-4f0b-a71d-e41d02e589d5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5847add22193d273cbe18ef95c4be2741a72265fe0db69c237b7a86b5ac15b51/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/465e1464aa13f2b03e5119bd5dd6ad2cc127bc7a0a8b68d14ef7e01e3d5ebe0a/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e85effca-b850-4af4-9086-ccfd965b7934/volumes/kubernetes.io~projected/kube-api-access-jwfgr DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ed974afc27d40236bc9889521d4ec285ca2b3450459909f787fa680be34fb41/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d839fc45-f2f4-4ce4-95d8-9e3145342973/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/875f6c48-91a9-4f0b-a71d-e41d02e589d5/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:326 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f092da451c9d7ac853e0529c3e2ed53ec04ddfc1e9918c52c86c293a711b6faf/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb7d909-e423-4efb-89ba-97cddc92628e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0/volumes/kubernetes.io~projected/kube-api-access-2skn4 DeviceMajor:0 DeviceMinor:453 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb967bfd-fb93-4051-bda5-9dbdf3589a69/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9009d30d-e890-4365-83af-2244164260f3/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e84786c71912f055085170107984e21bbd7d259bb63cbc30f96f46a18473d800/userdata/shm DeviceMajor:0 DeviceMinor:456 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85b7a4c9fffe7efecd43ab2ccecb1c9b8ee8b0432a9c97d0246b279322296d09/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01a49e9fa13dadb93adfb64727b600a7c5da3af980482e09cc09f40155e3186a/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e4f37ae-5582-442e-bf04-a2cc29932ec6/volumes/kubernetes.io~projected/kube-api-access-qqbwl DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/878dc633-6f09-450f-ab79-8788f14e3055/volumes/kubernetes.io~projected/kube-api-access-s2sm9 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3b7f9f9487f36b1 MacAddress:36:8f:f3:d5:81:07 Speed:10000 Mtu:8901} {Name:49c4e62507582f2 MacAddress:72:58:0c:82:f5:f9 Speed:10000 Mtu:8901} {Name:662efec249f9348 MacAddress:16:fe:7c:df:c2:d6 Speed:10000 Mtu:8901} {Name:912b8930f9f8f8e MacAddress:76:b4:d0:95:e6:1b Speed:10000 Mtu:8901} {Name:bafb3bf21c841d6 MacAddress:c2:0a:35:b9:63:5e Speed:10000 Mtu:8901} {Name:bb9f61588c747ca MacAddress:8a:e6:b5:a0:4e:cf Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:df:93:1a:ff Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:f2:8e:d7:ed:3f Speed:0 Mtu:8901} {Name:d4fdf7c93e52fcc MacAddress:f2:dd:e6:3e:92:07 Speed:10000 Mtu:8901} {Name:de8d654ac99108b MacAddress:9a:f6:cb:ce:a5:36 Speed:10000 Mtu:8901} {Name:e3eb10ad2da597c MacAddress:36:07:24:e0:77:7b Speed:10000 Mtu:8901} {Name:e6d7fa437c70919 MacAddress:92:50:f3:37:e3:81 Speed:10000 Mtu:8901} {Name:e84786c71912f05 MacAddress:6a:cf:23:9b:16:66 Speed:10000 Mtu:8901} {Name:eb075e522fd5313 MacAddress:ca:44:8b:23:7d:0b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:df:93:1a:ff Speed:0 Mtu:9001} {Name:f092da451c9d7ac MacAddress:4a:c4:7c:d8:e7:76 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:22:65:65:c3:89:28 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:98:90:80:e7:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:43:22.944697 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.944701 5736 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:43:22.944999 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.944757 5736 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:43:22.945076 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945040 5736 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:43:22.945203 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945078 5736 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:43:22.945248 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945205 5736 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:43:22.945248 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945213 5736 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:43:22.945248 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945232 5736 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:43:22.945248 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945245 5736 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:43:22.945393 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945308 5736 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:43:22.945492 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945482 5736 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:43:22.945635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945625 5736 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:43:22.945672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945663 5736 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:43:22.945707 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945676 5736 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:43:22.945707 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945686 5736 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:43:22.945707 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945694 5736 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:43:22.945707 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.945702 5736 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:43:22.946635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.946619 5736 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:43:22.946681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.946638 5736 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:43:22.947114 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.947100 5736 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:43:22.949657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.949642 5736 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:43:22.949704 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.949658 5736 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:43:22.949771 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.949762 5736 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 22:43:22.949942 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.949932 5736 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:43:22.949972 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.949960 5736 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:43:22.950063 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950054 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:43:22.950092 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950068 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:43:22.950092 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950074 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:43:22.950092 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950079 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:43:22.950092 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950085 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:43:22.950092 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950091 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:43:22.950219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950096 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:43:22.950219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950102 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:43:22.950219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950109 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:43:22.950219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950115 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:43:22.950219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950127 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:43:22.950219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950136 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:43:22.950219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950162 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:43:22.950219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.950168 5736 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:43:22.953360 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.953348 5736 server.go:1263] "Started kubelet" Oct 29 22:43:22.953454 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.953430 5736 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:43:22.953508 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.953421 5736 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:43:22.953508 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.953488 5736 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:43:22.953958 ip-10-0-143-172 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:43:22.954215 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.954192 5736 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:43:22.954597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.954566 5736 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:43:22.959496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.959477 5736 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.959613 5736 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.959621 5736 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.959641 5736 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:22.959638 5736 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-172.ec2.internal\" not found" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.959629 5736 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.959756 5736 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:37:13 +0000 UTC" deadline="2028-06-20 02:24:58.973990262 +0000 UTC" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.959775 5736 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14379h41m36.014218285s" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.961280 5736 factory.go:55] Registering systemd factory Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.961295 5736 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.961686 5736 factory.go:153] Registering CRI-O factory Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.961703 5736 factory.go:223] Registration of the crio container factory successfully Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.961759 5736 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.961779 5736 factory.go:103] Registering Raw factory Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.961796 5736 manager.go:1199] Started watching for new ooms in manager Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962065 5736 manager.go:319] Starting recovery of all containers Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962221 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5" volumeName="kubernetes.io/configmap/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-metrics-client-ca" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962246 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d839fc45-f2f4-4ce4-95d8-9e3145342973" volumeName="kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-config" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962254 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54a953c6-9194-404e-b8af-7ef3540d0e6d" volumeName="kubernetes.io/configmap/54a953c6-9194-404e-b8af-7ef3540d0e6d-iptables-alerter-script" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962261 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdf7ea9-aed7-4071-ada6-2f0518de3a87" volumeName="kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-tls" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962270 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-tls-assets" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962277 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e85effca-b850-4af4-9086-ccfd965b7934" volumeName="kubernetes.io/projected/e85effca-b850-4af4-9086-ccfd965b7934-kube-api-access-jwfgr" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962284 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb913afd-66ca-46e3-b9d1-66cff498245d" volumeName="kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962315 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="875f6c48-91a9-4f0b-a71d-e41d02e589d5" volumeName="kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-metrics-client-ca" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962325 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="878dc633-6f09-450f-ab79-8788f14e3055" volumeName="kubernetes.io/projected/878dc633-6f09-450f-ab79-8788f14e3055-kube-api-access-s2sm9" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962332 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962341 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1d25bfd-dba6-4268-8123-533f4b106c0a" volumeName="kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-metrics-client-ca" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962349 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b78e901b-2f4b-4ee5-a96d-573988c5cadd" volumeName="kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-binary-copy" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962358 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5" volumeName="kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962365 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b08b826d-232a-48be-ade2-dd833d8c3423" volumeName="kubernetes.io/projected/b08b826d-232a-48be-ade2-dd833d8c3423-kube-api-access-rrlvp" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962374 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31f1eee-73b6-4b6d-bde0-1006646b0173" volumeName="kubernetes.io/secret/d31f1eee-73b6-4b6d-bde0-1006646b0173-metrics-tls" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962387 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="875f6c48-91a9-4f0b-a71d-e41d02e589d5" volumeName="kubernetes.io/projected/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-api-access-p4kmz" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962396 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-cluster-tls-config" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962403 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1d25bfd-dba6-4268-8123-533f4b106c0a" volumeName="kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-tls" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962410 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5" volumeName="kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962419 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d839fc45-f2f4-4ce4-95d8-9e3145342973" volumeName="kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-script-lib" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962426 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d839fc45-f2f4-4ce4-95d8-9e3145342973" volumeName="kubernetes.io/secret/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962433 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e4f37ae-5582-442e-bf04-a2cc29932ec6" volumeName="kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-metrics-server-audit-profiles" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962441 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fd11cc5-35ab-4347-8914-e953b71bc322" volumeName="kubernetes.io/configmap/4fd11cc5-35ab-4347-8914-e953b71bc322-konnectivity-ca" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962448 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d4c20d8-f5e8-450b-b4f4-8199b277bbb0" volumeName="kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962464 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b726c8b-651c-4d5e-afa5-26a9f159d609" volumeName="kubernetes.io/empty-dir/8b726c8b-651c-4d5e-afa5-26a9f159d609-tmp-dir" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962471 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdf7ea9-aed7-4071-ada6-2f0518de3a87" volumeName="kubernetes.io/configmap/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-metrics-client-ca" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962479 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5" volumeName="kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962488 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5" volumeName="kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962500 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31f1eee-73b6-4b6d-bde0-1006646b0173" volumeName="kubernetes.io/empty-dir/d31f1eee-73b6-4b6d-bde0-1006646b0173-tmp-dir" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962509 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9009d30d-e890-4365-83af-2244164260f3" volumeName="kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-serving-certs-ca-bundle" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962517 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-kube-api-access-fngpp" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962526 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b78e901b-2f4b-4ee5-a96d-573988c5cadd" volumeName="kubernetes.io/projected/b78e901b-2f4b-4ee5-a96d-573988c5cadd-kube-api-access-rx6bq" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962533 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee899d2-5ec5-443d-b004-89ce10507f7d" volumeName="kubernetes.io/configmap/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962540 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d839fc45-f2f4-4ce4-95d8-9e3145342973" volumeName="kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-env-overrides" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962548 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e4f37ae-5582-442e-bf04-a2cc29932ec6" volumeName="kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962555 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e4f37ae-5582-442e-bf04-a2cc29932ec6" volumeName="kubernetes.io/empty-dir/1e4f37ae-5582-442e-bf04-a2cc29932ec6-audit-log" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962562 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e4f37ae-5582-442e-bf04-a2cc29932ec6" volumeName="kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962569 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="878dc633-6f09-450f-ab79-8788f14e3055" volumeName="kubernetes.io/configmap/878dc633-6f09-450f-ab79-8788f14e3055-serviceca" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962599 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdf7ea9-aed7-4071-ada6-2f0518de3a87" volumeName="kubernetes.io/projected/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-kube-api-access-rj97r" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962613 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9009d30d-e890-4365-83af-2244164260f3" volumeName="kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-metrics-client-ca" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962627 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a66d8310-cc78-4a48-bc21-03fa742d0829" volumeName="kubernetes.io/projected/a66d8310-cc78-4a48-bc21-03fa742d0829-kube-api-access-6kkhw" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962638 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5" volumeName="kubernetes.io/projected/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-kube-api-access-dfnkq" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962650 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1d25bfd-dba6-4268-8123-533f4b106c0a" volumeName="kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 29 22:43:22.962629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962660 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b78e901b-2f4b-4ee5-a96d-573988c5cadd" volumeName="kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962673 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31f1eee-73b6-4b6d-bde0-1006646b0173" volumeName="kubernetes.io/projected/d31f1eee-73b6-4b6d-bde0-1006646b0173-kube-api-access-9qt87" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962685 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0087fb0-f6d1-4f82-b78b-7f460f997ef0" volumeName="kubernetes.io/projected/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-kube-api-access-95k69" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962696 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9009d30d-e890-4365-83af-2244164260f3" volumeName="kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962706 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-metrics-client-ca" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962717 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54a953c6-9194-404e-b8af-7ef3540d0e6d" volumeName="kubernetes.io/projected/54a953c6-9194-404e-b8af-7ef3540d0e6d-kube-api-access-5x79c" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962728 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a6a57a5-af08-4ceb-a886-c176359b8d5a" volumeName="kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cni-binary-copy" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962738 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9009d30d-e890-4365-83af-2244164260f3" volumeName="kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-telemeter-client-tls" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962749 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb967bfd-fb93-4051-bda5-9dbdf3589a69" volumeName="kubernetes.io/projected/bb967bfd-fb93-4051-bda5-9dbdf3589a69-kube-api-access-w2gfd" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962762 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0087fb0-f6d1-4f82-b78b-7f460f997ef0" volumeName="kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-tuned" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962774 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e4f37ae-5582-442e-bf04-a2cc29932ec6" volumeName="kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-tls" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962785 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="875f6c48-91a9-4f0b-a71d-e41d02e589d5" volumeName="kubernetes.io/empty-dir/875f6c48-91a9-4f0b-a71d-e41d02e589d5-volume-directive-shadow" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962797 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9009d30d-e890-4365-83af-2244164260f3" volumeName="kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-federate-client-tls" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962818 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962830 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee899d2-5ec5-443d-b004-89ce10507f7d" volumeName="kubernetes.io/projected/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-api-access-g8xcj" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962841 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee899d2-5ec5-443d-b004-89ce10507f7d" volumeName="kubernetes.io/secret/cee899d2-5ec5-443d-b004-89ce10507f7d-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962852 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fd11cc5-35ab-4347-8914-e953b71bc322" volumeName="kubernetes.io/secret/4fd11cc5-35ab-4347-8914-e953b71bc322-agent-certs" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962864 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="875f6c48-91a9-4f0b-a71d-e41d02e589d5" volumeName="kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962878 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-main-db" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962890 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-config-out" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962902 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b78e901b-2f4b-4ee5-a96d-573988c5cadd" volumeName="kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962915 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee899d2-5ec5-443d-b004-89ce10507f7d" volumeName="kubernetes.io/empty-dir/cee899d2-5ec5-443d-b004-89ce10507f7d-data-volume" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962927 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31f1eee-73b6-4b6d-bde0-1006646b0173" volumeName="kubernetes.io/configmap/d31f1eee-73b6-4b6d-bde0-1006646b0173-config-volume" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962938 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d839fc45-f2f4-4ce4-95d8-9e3145342973" volumeName="kubernetes.io/projected/d839fc45-f2f4-4ce4-95d8-9e3145342973-kube-api-access-rtq46" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962950 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="875f6c48-91a9-4f0b-a71d-e41d02e589d5" volumeName="kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-tls" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962961 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b726c8b-651c-4d5e-afa5-26a9f159d609" volumeName="kubernetes.io/projected/8b726c8b-651c-4d5e-afa5-26a9f159d609-kube-api-access-vpt7k" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962973 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9009d30d-e890-4365-83af-2244164260f3" volumeName="kubernetes.io/projected/9009d30d-e890-4365-83af-2244164260f3-kube-api-access-wcqw6" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962986 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.962998 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963009 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb967bfd-fb93-4051-bda5-9dbdf3589a69" volumeName="kubernetes.io/secret/bb967bfd-fb93-4051-bda5-9dbdf3589a69-monitoring-plugin-cert" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963021 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5" volumeName="kubernetes.io/empty-dir/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-tmp" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963032 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5" volumeName="kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-grpc-tls" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963043 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e4f37ae-5582-442e-bf04-a2cc29932ec6" volumeName="kubernetes.io/projected/1e4f37ae-5582-442e-bf04-a2cc29932ec6-kube-api-access-qqbwl" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963055 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e4f37ae-5582-442e-bf04-a2cc29932ec6" volumeName="kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-client-ca-bundle" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963069 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5" volumeName="kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-tls" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963081 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e85effca-b850-4af4-9086-ccfd965b7934" volumeName="kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963092 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0087fb0-f6d1-4f82-b78b-7f460f997ef0" volumeName="kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-tmp" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963104 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b08b826d-232a-48be-ade2-dd833d8c3423" volumeName="kubernetes.io/secret/b08b826d-232a-48be-ade2-dd833d8c3423-cert" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963116 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a6a57a5-af08-4ceb-a886-c176359b8d5a" volumeName="kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-daemon-config" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963129 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9009d30d-e890-4365-83af-2244164260f3" volumeName="kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963143 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-config-volume" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963155 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-web-config" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963167 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb7d909-e423-4efb-89ba-97cddc92628e" volumeName="kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963180 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1d25bfd-dba6-4268-8123-533f4b106c0a" volumeName="kubernetes.io/projected/b1d25bfd-dba6-4268-8123-533f4b106c0a-kube-api-access-w58hz" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963193 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1d25bfd-dba6-4268-8123-533f4b106c0a" volumeName="kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963206 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1d25bfd-dba6-4268-8123-533f4b106c0a" volumeName="kubernetes.io/empty-dir/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-textfile" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963217 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a6a57a5-af08-4ceb-a886-c176359b8d5a" volumeName="kubernetes.io/projected/6a6a57a5-af08-4ceb-a886-c176359b8d5a-kube-api-access-lr776" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963228 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="875f6c48-91a9-4f0b-a71d-e41d02e589d5" volumeName="kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963240 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdf7ea9-aed7-4071-ada6-2f0518de3a87" volumeName="kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963252 5736 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9009d30d-e890-4365-83af-2244164260f3" volumeName="kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963265 5736 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:43:22.965386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.963281 5736 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:43:22.989967 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.989940 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:43:22.993749 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.993710 5736 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:43:22.995501 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.995475 5736 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:43:22.995626 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.995512 5736 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:43:22.995626 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.995534 5736 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:43:22.995626 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:22.995572 5736 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 22:43:23.000079 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:22.998795 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:43:23.005374 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.005291 5736 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:43:23.005942 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.005921 5736 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 22:43:23.007944 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.007908 5736 generic.go:356] "Generic (PLEG): container finished" podID="3d509ddddb3f91e54ef73594cf479892" containerID="7dd5c1c7c273402203bb71f30cb821b9395848265324e78f4b15dd78b225ce09" exitCode=0 Oct 29 22:43:23.012319 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.012299 5736 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:43:23.024371 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.024345 5736 generic.go:356] "Generic (PLEG): container finished" podID="9eb7d909-e423-4efb-89ba-97cddc92628e" containerID="184b92331e6d03d4aa342e3a323dc7bd191ee86dc6837d73fb89c21fa2a68e21" exitCode=0 Oct 29 22:43:23.026062 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.026043 5736 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:43:23.026156 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.026135 5736 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:43:23.026591 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.026550 5736 generic.go:356] "Generic (PLEG): container finished" podID="b1d25bfd-dba6-4268-8123-533f4b106c0a" containerID="641f9013df5c386a1f2fd590c4b60c309a1964ef2ff41458fc166b6e7a0c3251" exitCode=0 Oct 29 22:43:23.044080 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.044062 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:43:23.044406 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.044380 5736 generic.go:356] "Generic (PLEG): container finished" podID="d839fc45-f2f4-4ce4-95d8-9e3145342973" containerID="02a48f5861a7305427c0335950a2bfe7a55381b3582c1d7e83233968a255f7c1" exitCode=1 Oct 29 22:43:23.049452 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.049431 5736 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="439b8e91ad01d997eaf08639c131b68e5a1d1a8bd5d35ff197e11dcfae24e298" exitCode=0 Oct 29 22:43:23.049452 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.049449 5736 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="2a1ff3d8868ee7ec52f73f6554071db48e3f1fcb1b708d42536809688f509997" exitCode=0 Oct 29 22:43:23.049452 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.049455 5736 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="8ef08b5be091be9291f84caeb6229589864058ed2cf784a84b8575d083b279a2" exitCode=0 Oct 29 22:43:23.049657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.049461 5736 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="bfcf41f73e50404dab4980551ee4b5fc0d445cbeeef2d7169fdd5b1b04fe0f70" exitCode=0 Oct 29 22:43:23.049657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.049466 5736 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="7235dd1607ddeb64c05e2b32c5977006fcbb1cfde64e44efbd7db4bd73fc8132" exitCode=0 Oct 29 22:43:23.049657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.049472 5736 generic.go:356] "Generic (PLEG): container finished" podID="b78e901b-2f4b-4ee5-a96d-573988c5cadd" containerID="2df887541f355dde6d0f9978107aafa687f65bf18541766b29d1551cf15c1121" exitCode=0 Oct 29 22:43:23.080640 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.080555 5736 manager.go:324] Recovery completed Oct 29 22:43:23.095871 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:23.095847 5736 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 29 22:43:23.103645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103625 5736 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:43:23.103645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103640 5736 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:43:23.103808 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103655 5736 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:43:23.103808 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103769 5736 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 22:43:23.103808 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103780 5736 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 22:43:23.103808 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103796 5736 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 22:43:23.103808 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103801 5736 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 22:43:23.103808 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103807 5736 policy_none.go:50] "Start" Oct 29 22:43:23.103989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103816 5736 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:43:23.103989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103824 5736 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:43:23.103989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103942 5736 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 22:43:23.103989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103948 5736 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 22:43:23.103989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.103954 5736 policy_none.go:44] "Start" Oct 29 22:43:23.108439 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.108413 5736 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:43:23.113085 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.113070 5736 manager.go:341] "Starting Device Plugin manager" Oct 29 22:43:23.113555 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:23.113099 5736 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:43:23.113659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.113556 5736 server.go:87] "Starting device plugin registration server" Oct 29 22:43:23.113869 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.113856 5736 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:43:23.113931 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.113871 5736 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:43:23.113982 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.113952 5736 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:43:23.114694 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.114066 5736 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:43:23.114694 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.114114 5736 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:43:23.114694 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.114121 5736 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:43:23.114694 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.114163 5736 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:43:23.114099508Z","UUID":"e31effcd-358b-487a-9c45-8d2c09c02cfb","Handler":null,"Name":"","Endpoint":""} Oct 29 22:43:23.115424 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.115401 5736 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:43:23.115424 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.115424 5736 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:43:23.115929 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:23.115913 5736 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:43:23.213970 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.213931 5736 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:43:23.215690 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.215672 5736 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:43:23.215758 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.215703 5736 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:43:23.215758 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.215712 5736 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-172.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:43:23.215846 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.215759 5736 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.247051 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.247032 5736 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.247136 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.247087 5736 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.296244 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.296202 5736 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal"] Oct 29 22:43:23.296816 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.296785 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" event={"ID":"f3d32c15ddf0c3d5f6df2743c0d67b79","Type":"ContainerStarted","Data":"5253c13d578973165ea4728d14e83bc389dd35f206d21f869bdf1eddc93b1b0b"} Oct 29 22:43:23.296875 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.296823 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" event={"ID":"f3d32c15ddf0c3d5f6df2743c0d67b79","Type":"ContainerStarted","Data":"5847add22193d273cbe18ef95c4be2741a72265fe0db69c237b7a86b5ac15b51"} Oct 29 22:43:23.296875 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.296837 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" event={"ID":"3d509ddddb3f91e54ef73594cf479892","Type":"ContainerStarted","Data":"4477b3bfe077fba351aa56281f42131d92051d515246b0e0fef732ff9e52c4d7"} Oct 29 22:43:23.296875 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.296844 5736 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.296985 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.296849 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" event={"ID":"3d509ddddb3f91e54ef73594cf479892","Type":"ContainerDied","Data":"7dd5c1c7c273402203bb71f30cb821b9395848265324e78f4b15dd78b225ce09"} Oct 29 22:43:23.297049 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.296973 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" event={"ID":"3d509ddddb3f91e54ef73594cf479892","Type":"ContainerStarted","Data":"f7a8fd2af6af21a4139d5b7c6d92fd2224131579fcc42e1784ae607e72b93edc"} Oct 29 22:43:23.297049 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.297011 5736 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.319048 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.319027 5736 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:43:23.319126 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:23.319071 5736 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-172.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.319796 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.319783 5736 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:43:23.319869 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:23.319816 5736 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.408152 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.408084 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.408152 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.408128 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3d32c15ddf0c3d5f6df2743c0d67b79-config\") pod \"kube-apiserver-proxy-ip-10-0-143-172.ec2.internal\" (UID: \"f3d32c15ddf0c3d5f6df2743c0d67b79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.408152 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.408153 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.508589 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.508562 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.508678 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.508599 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.508678 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.508620 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3d32c15ddf0c3d5f6df2743c0d67b79-config\") pod \"kube-apiserver-proxy-ip-10-0-143-172.ec2.internal\" (UID: \"f3d32c15ddf0c3d5f6df2743c0d67b79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.508678 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.508666 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.508780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.508672 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d509ddddb3f91e54ef73594cf479892-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" (UID: \"3d509ddddb3f91e54ef73594cf479892\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.508780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.508667 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3d32c15ddf0c3d5f6df2743c0d67b79-config\") pod \"kube-apiserver-proxy-ip-10-0-143-172.ec2.internal\" (UID: \"f3d32c15ddf0c3d5f6df2743c0d67b79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal" Oct 29 22:43:23.946188 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.946157 5736 apiserver.go:52] "Watching apiserver" Oct 29 22:43:23.992184 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.992161 5736 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:43:23.992765 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:23.992744 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-ncjz6","openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf","kube-system/konnectivity-agent-lgqrb","openshift-dns/node-resolver-5rgg6","openshift-multus/multus-cpp5c","openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7","openshift-network-operator/iptables-alerter-dcjkg","openshift-insights/insights-runtime-extractor-zcdnt","openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb","openshift-monitoring/telemeter-client-76865c6696-5hm7s","kube-system/global-pull-secret-syncer-twb2h","openshift-cluster-node-tuning-operator/tuned-jk6th","openshift-ingress-canary/ingress-canary-tvxzw","openshift-dns/dns-default-q7trl","openshift-ovn-kubernetes/ovnkube-node-vppzr","openshift-monitoring/thanos-querier-86c4948c74-qwn7r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp","openshift-network-diagnostics/network-check-target-lglpm","openshift-monitoring/metrics-server-75f497d78f-wzp7l","openshift-multus/network-metrics-daemon-d286w","openshift-monitoring/node-exporter-rc9cg","openshift-multus/multus-additional-cni-plugins-ptfk2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal","openshift-monitoring/alertmanager-main-0","kube-system/kube-apiserver-proxy-ip-10-0-143-172.ec2.internal"] Oct 29 22:43:24.008010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.007989 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.008173 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.008156 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:43:24.008234 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.008159 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.009171 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009149 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.009281 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009266 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:43:24.009341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009280 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.009341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009309 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:43:24.009341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009323 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.009452 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009358 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.009452 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009367 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:43:24.009452 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009425 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.009627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009495 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.009669 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009646 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.009706 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009667 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:43:24.009735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.009703 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.010233 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.010193 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:43:24.010344 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.010329 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fchfg\"" Oct 29 22:43:24.010752 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.010730 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-phcjr\"" Oct 29 22:43:24.010868 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.010755 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m52c5\"" Oct 29 22:43:24.010868 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.010848 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-28brx\"" Oct 29 22:43:24.010962 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.010915 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tj892\"" Oct 29 22:43:24.011036 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.011020 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7jjw\"" Oct 29 22:43:24.011230 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.011215 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:43:24.020595 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.020562 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kfq4\"" Oct 29 22:43:24.020989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.020970 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:43:24.021074 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.021013 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:43:24.021144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.021085 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.021144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.021101 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.022826 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.022808 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:43:24.022889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.022858 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.023356 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.023338 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:43:24.023456 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.023442 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rpfjl\"" Oct 29 22:43:24.023508 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.023487 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.023508 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.023501 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:43:24.023716 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.023699 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.024541 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024513 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.024713 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024695 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d6hqb\"" Oct 29 22:43:24.024775 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024766 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.024837 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024767 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:43:24.024837 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024812 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:43:24.024936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024832 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:43:24.024936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024847 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:43:24.024936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024834 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.024936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024862 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:43:24.024936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024884 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vllbq\"" Oct 29 22:43:24.024936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024906 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:43:24.024936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024924 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.025195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024947 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ltz9s\"" Oct 29 22:43:24.025195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024970 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:43:24.025195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.024997 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:43:24.025195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.025013 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xjmd7\"" Oct 29 22:43:24.025195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.025029 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:43:24.025195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.025054 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:43:24.025195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.025089 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:43:24.025195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.025189 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:43:24.025469 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.025241 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:43:24.025469 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.025312 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 29 22:43:24.025692 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.025678 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 29 22:43:24.026116 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.026101 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 29 22:43:24.026245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.026117 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:43:24.026290 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.026247 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.026326 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.026308 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:43:24.026374 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.026341 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 22:43:24.032082 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.032065 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 22:43:24.036265 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036244 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 29 22:43:24.036370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036334 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nqp65\"" Oct 29 22:43:24.036657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036640 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 29 22:43:24.036726 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036697 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 29 22:43:24.036776 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036759 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 29 22:43:24.036776 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036769 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 22:43:24.036876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036774 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 22:43:24.036876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036840 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 29 22:43:24.036876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036868 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 22:43:24.037026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036876 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 29 22:43:24.037026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036975 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s2zp5\"" Oct 29 22:43:24.037026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036888 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 29 22:43:24.037026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036900 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 29 22:43:24.037026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036914 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 22:43:24.037026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036917 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 29 22:43:24.037026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036870 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x4fl8\"" Oct 29 22:43:24.037026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036925 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mcv8v\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037030 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8xd28\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036936 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036938 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f88xp\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036943 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036969 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037060 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036910 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036986 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.036967 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037102 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ljgzz\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037188 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2plp\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037277 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-62qeduce6lfgh\"" Oct 29 22:43:24.037283 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037288 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 29 22:43:24.037621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037312 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 29 22:43:24.037621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037343 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 29 22:43:24.037621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037350 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nxz4d\"" Oct 29 22:43:24.037621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037412 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dtoi9vcgc088c\"" Oct 29 22:43:24.037621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037476 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-98j5z\"" Oct 29 22:43:24.037621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037507 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xvchs\"" Oct 29 22:43:24.037621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.037554 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 29 22:43:24.061834 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.061815 5736 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:43:24.062407 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.062389 5736 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:24.085034 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.085018 5736 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:43:24.085130 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:24.085056 5736 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-172.ec2.internal" Oct 29 22:43:24.111066 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111044 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:43:24.111167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111083 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.111167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111106 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.111167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111129 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb967bfd-fb93-4051-bda5-9dbdf3589a69-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:24.111167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111158 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-sys-fs\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.111376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111246 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d31f1eee-73b6-4b6d-bde0-1006646b0173-metrics-tls\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:24.111376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111293 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.111376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111334 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54a953c6-9194-404e-b8af-7ef3540d0e6d-host-slash\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:43:24.111376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111356 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.111376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111356 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.111559 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111358 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-system-cni-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.111559 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111425 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb967bfd-fb93-4051-bda5-9dbdf3589a69-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:24.111559 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111486 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-conf\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.111559 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111497 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d31f1eee-73b6-4b6d-bde0-1006646b0173-metrics-tls\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:24.111559 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111532 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.111757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111613 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.111757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111627 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-federate-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.111757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111703 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.111757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111728 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-var-lib-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.111757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111733 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.111924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111760 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cee899d2-5ec5-443d-b004-89ce10507f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.111924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111786 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-federate-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.111924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111826 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-system-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.111924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111855 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-socket-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.111924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111877 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-device-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.112081 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111933 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11cc5-35ab-4347-8914-e953b71bc322-konnectivity-ca\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:43:24.112081 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111940 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cee899d2-5ec5-443d-b004-89ce10507f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.112081 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.111995 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngpp\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-kube-api-access-fngpp\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.112081 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112024 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2gfd\" (UniqueName: \"kubernetes.io/projected/bb967bfd-fb93-4051-bda5-9dbdf3589a69-kube-api-access-w2gfd\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:24.112081 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112050 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.112081 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112078 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.112336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112101 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11cc5-35ab-4347-8914-e953b71bc322-konnectivity-ca\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:43:24.112336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112105 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.112336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112146 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qt87\" (UniqueName: \"kubernetes.io/projected/d31f1eee-73b6-4b6d-bde0-1006646b0173-kube-api-access-9qt87\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:24.112336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112173 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-systemd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.112336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112218 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrlvp\" (UniqueName: \"kubernetes.io/projected/b08b826d-232a-48be-ade2-dd833d8c3423-kube-api-access-rrlvp\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:43:24.112336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112243 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.112336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112254 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.112336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112266 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-multus\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.112336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112304 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.112625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112379 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.112625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112412 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-dbus\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:24.112625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112512 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-telemeter-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.112625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112532 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-env-overrides\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.112625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112569 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d31f1eee-73b6-4b6d-bde0-1006646b0173-config-volume\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:24.112625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112609 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-tmp\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.112794 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112626 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-tls\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.112794 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112693 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.112794 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112712 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.112794 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112738 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.112794 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112754 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.112930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112813 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kmz\" (UniqueName: \"kubernetes.io/projected/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-api-access-p4kmz\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.112930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112836 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-bin\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.112930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112754 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.112930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112888 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-config\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.112930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112913 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:24.112930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112928 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-run\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.113091 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112930 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d31f1eee-73b6-4b6d-bde0-1006646b0173-config-volume\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:24.113091 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.112944 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.113154 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.113104 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.113187 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.113159 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-tls\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.113187 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.113165 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-tmp\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.113376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.113329 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-env-overrides\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.113285 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-metrics-client-ca\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.113725 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d31f1eee-73b6-4b6d-bde0-1006646b0173-tmp-dir\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.113768 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-config-out\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.113878 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.113999 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114031 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114014 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d31f1eee-73b6-4b6d-bde0-1006646b0173-tmp-dir\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114050 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-telemeter-client-tls\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114045 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-metrics-client-ca\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114094 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb913afd-66ca-46e3-b9d1-66cff498245d-original-pull-secret\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114177 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-config-out\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114093 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114198 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114201 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9eb7d909-e423-4efb-89ba-97cddc92628e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114298 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-config\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114389 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-metrics-client-ca\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114662 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114727 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx6bq\" (UniqueName: \"kubernetes.io/projected/b78e901b-2f4b-4ee5-a96d-573988c5cadd-kube-api-access-rx6bq\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114729 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-metrics-client-ca\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114801 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/875f6c48-91a9-4f0b-a71d-e41d02e589d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114855 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj97r\" (UniqueName: \"kubernetes.io/projected/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-kube-api-access-rj97r\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114922 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cnibin\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114959 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.114992 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-kubernetes\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115022 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115085 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-k8s-cni-cncf-io\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115115 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-bin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115138 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysconfig\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115170 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115200 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-registration-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115231 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115261 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-var-lib-kubelet\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115354 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115389 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kkhw\" (UniqueName: \"kubernetes.io/projected/a66d8310-cc78-4a48-bc21-03fa742d0829-kube-api-access-6kkhw\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115429 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-socket-dir-parent\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115462 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr776\" (UniqueName: \"kubernetes.io/projected/6a6a57a5-af08-4ceb-a886-c176359b8d5a-kube-api-access-lr776\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115493 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfnkq\" (UniqueName: \"kubernetes.io/projected/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-kube-api-access-dfnkq\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115522 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfgr\" (UniqueName: \"kubernetes.io/projected/e85effca-b850-4af4-9086-ccfd965b7934-kube-api-access-jwfgr\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115552 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-slash\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115603 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fd11cc5-35ab-4347-8914-e953b71bc322-agent-certs\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115656 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cee899d2-5ec5-443d-b004-89ce10507f7d-data-volume\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115685 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-hostroot\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115711 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-wtmp\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115742 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-grpc-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115773 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-kubelet\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115935 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08b826d-232a-48be-ade2-dd833d8c3423-cert\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.115963 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cee899d2-5ec5-443d-b004-89ce10507f7d-crio-socket\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116005 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-daemon-config\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116030 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-host\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116062 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116099 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-node-log\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116128 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-log-socket\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116159 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtq46\" (UniqueName: \"kubernetes.io/projected/d839fc45-f2f4-4ce4-95d8-9e3145342973-kube-api-access-rtq46\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116184 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b726c8b-651c-4d5e-afa5-26a9f159d609-tmp-dir\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116213 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58hz\" (UniqueName: \"kubernetes.io/projected/b1d25bfd-dba6-4268-8123-533f4b106c0a-kube-api-access-w58hz\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116277 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116333 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-lib-modules\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116364 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95k69\" (UniqueName: \"kubernetes.io/projected/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-kube-api-access-95k69\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116415 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-metrics-server-audit-profiles\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116446 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-etc-kubernetes\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116479 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-tls\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116511 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-kubelet-config\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116545 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-textfile\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116570 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-sys\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116619 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-netd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116652 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116684 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-metrics-client-ca\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116711 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.116985 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-textfile\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.117466 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/875f6c48-91a9-4f0b-a71d-e41d02e589d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.117688 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.117925 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.118348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.118106 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.118624 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-grpc-tls\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.118716 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cee899d2-5ec5-443d-b004-89ce10507f7d-data-volume\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.118813 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqw6\" (UniqueName: \"kubernetes.io/projected/9009d30d-e890-4365-83af-2244164260f3-kube-api-access-wcqw6\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.118903 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.118923 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fd11cc5-35ab-4347-8914-e953b71bc322-agent-certs\") pod \"konnectivity-agent-lgqrb\" (UID: \"4fd11cc5-35ab-4347-8914-e953b71bc322\") " pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.118943 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119006 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-netns\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119121 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-daemon-config\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119244 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119295 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-root\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119359 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbwl\" (UniqueName: \"kubernetes.io/projected/1e4f37ae-5582-442e-bf04-a2cc29932ec6-kube-api-access-qqbwl\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119397 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/878dc633-6f09-450f-ab79-8788f14e3055-host\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119436 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2sm9\" (UniqueName: \"kubernetes.io/projected/878dc633-6f09-450f-ab79-8788f14e3055-kube-api-access-s2sm9\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119559 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-tls\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119516 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119756 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xcj\" (UniqueName: \"kubernetes.io/projected/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-api-access-g8xcj\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119776 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85effca-b850-4af4-9086-ccfd965b7934-metrics-certs\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119799 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.119948 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b726c8b-651c-4d5e-afa5-26a9f159d609-tmp-dir\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120006 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120027 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120151 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-systemd-units\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120167 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120200 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovn-node-metrics-cert\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120237 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/878dc633-6f09-450f-ab79-8788f14e3055-serviceca\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120359 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120485 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120510 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-conf-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120530 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-multus-certs\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120551 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e4f37ae-5582-442e-bf04-a2cc29932ec6-audit-log\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120572 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b726c8b-651c-4d5e-afa5-26a9f159d609-hosts-file\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120645 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cni-binary-copy\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120953 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1e4f37ae-5582-442e-bf04-a2cc29932ec6-audit-log\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120979 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/878dc633-6f09-450f-ab79-8788f14e3055-serviceca\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121007 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08b826d-232a-48be-ade2-dd833d8c3423-cert\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121007 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cni-binary-copy\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.120995 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9009d30d-e890-4365-83af-2244164260f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121022 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovn-node-metrics-cert\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121064 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121209 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1e4f37ae-5582-442e-bf04-a2cc29932ec6-metrics-server-audit-profiles\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121250 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.121650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121368 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121753 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1d25bfd-dba6-4268-8123-533f4b106c0a-metrics-client-ca\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121788 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-modprobe-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121901 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121948 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.121966 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122022 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-netns\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122088 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpt7k\" (UniqueName: \"kubernetes.io/projected/8b726c8b-651c-4d5e-afa5-26a9f159d609-kube-api-access-vpt7k\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122132 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122216 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122327 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-kubelet\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122364 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122383 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122400 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-sys\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122496 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-tuned\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122533 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-ovn\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122544 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b78e901b-2f4b-4ee5-a96d-573988c5cadd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122597 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-tuned\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122646 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x79c\" (UniqueName: \"kubernetes.io/projected/54a953c6-9194-404e-b8af-7ef3540d0e6d-kube-api-access-5x79c\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122666 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-systemd\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122746 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122790 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-script-lib\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122844 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122950 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-web-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.122991 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d839fc45-f2f4-4ce4-95d8-9e3145342973-ovnkube-script-lib\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123035 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123088 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cnibin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123125 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.123128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123128 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-web-config\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123152 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-client-certs\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123160 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123258 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-client-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123283 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123298 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-serving-certs-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123387 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-config-volume\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123401 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-secret-metrics-server-client-certs\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123422 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54a953c6-9194-404e-b8af-7ef3540d0e6d-iptables-alerter-script\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123475 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1e4f37ae-5582-442e-bf04-a2cc29932ec6-client-ca-bundle\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123499 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123549 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9009d30d-e890-4365-83af-2244164260f3-serving-certs-ca-bundle\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123562 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9eb7d909-e423-4efb-89ba-97cddc92628e-config-volume\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123574 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-tmp\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123640 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54a953c6-9194-404e-b8af-7ef3540d0e6d-iptables-alerter-script\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123653 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-etc-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.124047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.123685 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-tmp\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.132216 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.132200 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrlvp\" (UniqueName: \"kubernetes.io/projected/b08b826d-232a-48be-ade2-dd833d8c3423-kube-api-access-rrlvp\") pod \"ingress-canary-tvxzw\" (UID: \"b08b826d-232a-48be-ade2-dd833d8c3423\") " pod="openshift-ingress-canary/ingress-canary-tvxzw" Oct 29 22:43:24.132272 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.132209 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2gfd\" (UniqueName: \"kubernetes.io/projected/bb967bfd-fb93-4051-bda5-9dbdf3589a69-kube-api-access-w2gfd\") pod \"monitoring-plugin-77f4f76594-4l2vb\" (UID: \"bb967bfd-fb93-4051-bda5-9dbdf3589a69\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:24.132830 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.132815 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngpp\" (UniqueName: \"kubernetes.io/projected/9eb7d909-e423-4efb-89ba-97cddc92628e-kube-api-access-fngpp\") pod \"alertmanager-main-0\" (UID: \"9eb7d909-e423-4efb-89ba-97cddc92628e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 22:43:24.132974 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.132960 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skn4\" (UniqueName: \"kubernetes.io/projected/7d4c20d8-f5e8-450b-b4f4-8199b277bbb0-kube-api-access-2skn4\") pod \"network-check-target-lglpm\" (UID: \"7d4c20d8-f5e8-450b-b4f4-8199b277bbb0\") " pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:43:24.134985 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.134967 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qt87\" (UniqueName: \"kubernetes.io/projected/d31f1eee-73b6-4b6d-bde0-1006646b0173-kube-api-access-9qt87\") pod \"dns-default-q7trl\" (UID: \"d31f1eee-73b6-4b6d-bde0-1006646b0173\") " pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:24.138145 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.138127 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kmz\" (UniqueName: \"kubernetes.io/projected/875f6c48-91a9-4f0b-a71d-e41d02e589d5-kube-api-access-p4kmz\") pod \"kube-state-metrics-5dc8d7d89b-7lhcf\" (UID: \"875f6c48-91a9-4f0b-a71d-e41d02e589d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7lhcf" Oct 29 22:43:24.144530 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.144497 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfgr\" (UniqueName: \"kubernetes.io/projected/e85effca-b850-4af4-9086-ccfd965b7934-kube-api-access-jwfgr\") pod \"network-metrics-daemon-d286w\" (UID: \"e85effca-b850-4af4-9086-ccfd965b7934\") " pod="openshift-multus/network-metrics-daemon-d286w" Oct 29 22:43:24.144986 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.144933 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtq46\" (UniqueName: \"kubernetes.io/projected/d839fc45-f2f4-4ce4-95d8-9e3145342973-kube-api-access-rtq46\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.145103 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.145043 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95k69\" (UniqueName: \"kubernetes.io/projected/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-kube-api-access-95k69\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.145199 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.145121 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbwl\" (UniqueName: \"kubernetes.io/projected/1e4f37ae-5582-442e-bf04-a2cc29932ec6-kube-api-access-qqbwl\") pod \"metrics-server-75f497d78f-wzp7l\" (UID: \"1e4f37ae-5582-442e-bf04-a2cc29932ec6\") " pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:24.145260 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.145194 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kkhw\" (UniqueName: \"kubernetes.io/projected/a66d8310-cc78-4a48-bc21-03fa742d0829-kube-api-access-6kkhw\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.145260 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.145237 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xcj\" (UniqueName: \"kubernetes.io/projected/cee899d2-5ec5-443d-b004-89ce10507f7d-kube-api-access-g8xcj\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.145321 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.145238 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj97r\" (UniqueName: \"kubernetes.io/projected/8cdf7ea9-aed7-4071-ada6-2f0518de3a87-kube-api-access-rj97r\") pod \"openshift-state-metrics-7bd7489957-zx4x7\" (UID: \"8cdf7ea9-aed7-4071-ada6-2f0518de3a87\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-zx4x7" Oct 29 22:43:24.145350 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.145330 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqw6\" (UniqueName: \"kubernetes.io/projected/9009d30d-e890-4365-83af-2244164260f3-kube-api-access-wcqw6\") pod \"telemeter-client-76865c6696-5hm7s\" (UID: \"9009d30d-e890-4365-83af-2244164260f3\") " pod="openshift-monitoring/telemeter-client-76865c6696-5hm7s" Oct 29 22:43:24.145393 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.145375 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpt7k\" (UniqueName: \"kubernetes.io/projected/8b726c8b-651c-4d5e-afa5-26a9f159d609-kube-api-access-vpt7k\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:43:24.145901 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.145882 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2sm9\" (UniqueName: \"kubernetes.io/projected/878dc633-6f09-450f-ab79-8788f14e3055-kube-api-access-s2sm9\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:43:24.148883 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.148867 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx6bq\" (UniqueName: \"kubernetes.io/projected/b78e901b-2f4b-4ee5-a96d-573988c5cadd-kube-api-access-rx6bq\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.149678 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.149660 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58hz\" (UniqueName: \"kubernetes.io/projected/b1d25bfd-dba6-4268-8123-533f4b106c0a-kube-api-access-w58hz\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.149844 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.149829 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr776\" (UniqueName: \"kubernetes.io/projected/6a6a57a5-af08-4ceb-a886-c176359b8d5a-kube-api-access-lr776\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.149881 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.149871 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfnkq\" (UniqueName: \"kubernetes.io/projected/c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5-kube-api-access-dfnkq\") pod \"thanos-querier-86c4948c74-qwn7r\" (UID: \"c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5\") " pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:24.151174 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.151157 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x79c\" (UniqueName: \"kubernetes.io/projected/54a953c6-9194-404e-b8af-7ef3540d0e6d-kube-api-access-5x79c\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:43:24.224452 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224431 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.224529 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224455 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-conf-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.224529 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224471 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-multus-certs\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.224529 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224486 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b726c8b-651c-4d5e-afa5-26a9f159d609-hosts-file\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:43:24.224529 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224500 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-modprobe-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.224529 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224514 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224533 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-netns\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224543 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-multus-certs\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224557 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-kubelet\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224562 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224543 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-conf-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224608 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-kubelet\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224614 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b726c8b-651c-4d5e-afa5-26a9f159d609-hosts-file\") pod \"node-resolver-5rgg6\" (UID: \"8b726c8b-651c-4d5e-afa5-26a9f159d609\") " pod="openshift-dns/node-resolver-5rgg6" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224610 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-sys\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224640 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-sys\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224653 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-ovn\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224662 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-modprobe-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224672 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-d\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224668 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-run-netns\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224701 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-ovn\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224676 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-systemd\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224729 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cnibin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224733 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-systemd\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224747 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.224778 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224768 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-etc-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224788 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-cnibin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224796 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-sys-fs\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224813 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54a953c6-9194-404e-b8af-7ef3540d0e6d-host-slash\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224824 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224833 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-system-cni-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224838 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-etc-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224848 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-conf\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224855 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54a953c6-9194-404e-b8af-7ef3540d0e6d-host-slash\") pod \"iptables-alerter-dcjkg\" (UID: \"54a953c6-9194-404e-b8af-7ef3540d0e6d\") " pod="openshift-network-operator/iptables-alerter-dcjkg" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224856 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-sys-fs\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224876 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224881 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-system-cni-dir\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224922 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-var-lib-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224933 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224940 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysctl-conf\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224950 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-var-lib-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224959 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-system-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.224980 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-socket-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225003 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-device-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225012 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-system-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225030 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-systemd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225058 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-device-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225056 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-multus\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225092 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-dbus\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225097 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-systemd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225060 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-socket-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225112 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-multus\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225134 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225161 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-bin\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225166 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-dbus\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225185 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-run\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225207 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cnibin\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225215 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225230 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-kubernetes\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225230 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-bin\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225257 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-run\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225259 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225256 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b78e901b-2f4b-4ee5-a96d-573988c5cadd-cnibin\") pod \"multus-additional-cni-plugins-ptfk2\" (UID: \"b78e901b-2f4b-4ee5-a96d-573988c5cadd\") " pod="openshift-multus/multus-additional-cni-plugins-ptfk2" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225283 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-k8s-cni-cncf-io\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.225277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225292 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-kubernetes\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225308 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-bin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225316 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-run-openvswitch\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225329 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-k8s-cni-cncf-io\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225330 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysconfig\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225351 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-var-lib-cni-bin\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225368 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-etc-sysconfig\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225368 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-registration-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225398 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225406 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-registration-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225421 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-var-lib-kubelet\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225444 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-socket-dir-parent\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225463 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-cni-dir\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225461 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-slash\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225485 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-var-lib-kubelet\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225492 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-slash\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225500 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-hostroot\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225503 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-multus-socket-dir-parent\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225524 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-wtmp\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225539 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-kubelet\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225562 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cee899d2-5ec5-443d-b004-89ce10507f7d-crio-socket\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225600 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-host\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225603 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-node-exporter-wtmp\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225599 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-kubelet\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225625 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-node-log\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225641 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-log-socket\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225645 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-node-log\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225626 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-hostroot\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225673 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-host\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225678 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-lib-modules\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225641 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cee899d2-5ec5-443d-b004-89ce10507f7d-crio-socket\") pod \"insights-runtime-extractor-zcdnt\" (UID: \"cee899d2-5ec5-443d-b004-89ce10507f7d\") " pod="openshift-insights/insights-runtime-extractor-zcdnt" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225695 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-log-socket\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225760 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-lib-modules\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225786 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-etc-kubernetes\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225803 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-kubelet-config\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225822 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-etc-kubernetes\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225824 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-sys\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225845 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-netd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225856 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb913afd-66ca-46e3-b9d1-66cff498245d-kubelet-config\") pod \"global-pull-secret-syncer-twb2h\" (UID: \"fb913afd-66ca-46e3-b9d1-66cff498245d\") " pod="kube-system/global-pull-secret-syncer-twb2h" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225861 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225881 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0087fb0-f6d1-4f82-b78b-7f460f997ef0-sys\") pod \"tuned-jk6th\" (UID: \"f0087fb0-f6d1-4f82-b78b-7f460f997ef0\") " pod="openshift-cluster-node-tuning-operator/tuned-jk6th" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225881 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-netns\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225906 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-host-cni-netd\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225905 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a6a57a5-af08-4ceb-a886-c176359b8d5a-host-run-netns\") pod \"multus-cpp5c\" (UID: \"6a6a57a5-af08-4ceb-a886-c176359b8d5a\") " pod="openshift-multus/multus-cpp5c" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225924 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a66d8310-cc78-4a48-bc21-03fa742d0829-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdjrp\" (UID: \"a66d8310-cc78-4a48-bc21-03fa742d0829\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdjrp" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225930 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-root\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225952 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/878dc633-6f09-450f-ab79-8788f14e3055-host\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225978 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-systemd-units\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225982 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1d25bfd-dba6-4268-8123-533f4b106c0a-root\") pod \"node-exporter-rc9cg\" (UID: \"b1d25bfd-dba6-4268-8123-533f4b106c0a\") " pod="openshift-monitoring/node-exporter-rc9cg" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.225987 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/878dc633-6f09-450f-ab79-8788f14e3055-host\") pod \"node-ca-ncjz6\" (UID: \"878dc633-6f09-450f-ab79-8788f14e3055\") " pod="openshift-image-registry/node-ca-ncjz6" Oct 29 22:43:24.226378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:24.226031 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d839fc45-f2f4-4ce4-95d8-9e3145342973-systemd-units\") pod \"ovnkube-node-vppzr\" (UID: \"d839fc45-f2f4-4ce4-95d8-9e3145342973\") " pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:25.472315 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:25.472287 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86c4948c74-qwn7r" Oct 29 22:43:26.476864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:26.476833 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4l2vb" Oct 29 22:43:26.667256 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:26.667228 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q7trl" Oct 29 22:43:26.822010 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:26.821940 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:43:30.794048 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:30.794009 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:43:32.588419 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:32.588390 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:33.222278 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:33.222250 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:33.222446 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:33.222436 5736 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:43:42.274263 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:42.274193 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-75f497d78f-wzp7l" Oct 29 22:43:43.667981 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:43.667954 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vppzr" Oct 29 22:43:49.285476 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:43:49.285441 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lgqrb" Oct 29 22:43:56.821859 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:43:56.821822 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:44:00.797341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:44:00.797311 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lglpm" Oct 29 22:44:26.821694 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:44:26.821658 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:44:56.822131 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:44:56.822091 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:45:26.821726 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:45:26.821644 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:45:40.815132 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.815099 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs"] Oct 29 22:45:40.817594 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.817562 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:40.831875 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.831855 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-g6z8r\"" Oct 29 22:45:40.831980 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.831901 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 29 22:45:40.832045 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.832024 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 29 22:45:40.843091 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.843064 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs"] Oct 29 22:45:40.898810 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.898783 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rd5\" (UniqueName: \"kubernetes.io/projected/24907517-a224-4ef7-910d-e975d9f2c517-kube-api-access-f4rd5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:40.898948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.898823 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:40.898948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.898851 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:40.912359 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.912338 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8"] Oct 29 22:45:40.914767 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.914747 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:40.935166 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.935143 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-rvjfw\"" Oct 29 22:45:40.938935 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.938913 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8"] Oct 29 22:45:40.999482 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.999455 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rd5\" (UniqueName: \"kubernetes.io/projected/24907517-a224-4ef7-910d-e975d9f2c517-kube-api-access-f4rd5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:40.999672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.999493 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:40.999672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.999532 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4xj\" (UniqueName: \"kubernetes.io/projected/0495b7c6-52cf-411c-bc40-d210cd192855-kube-api-access-9r4xj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:40.999672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.999550 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:40.999672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.999588 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:40.999672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:40.999614 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:41.000024 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.000000 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:41.000024 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.000015 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:41.018982 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.018956 5736 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:45:41.021184 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.021162 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rd5\" (UniqueName: \"kubernetes.io/projected/24907517-a224-4ef7-910d-e975d9f2c517-kube-api-access-f4rd5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:41.100706 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.100633 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4xj\" (UniqueName: \"kubernetes.io/projected/0495b7c6-52cf-411c-bc40-d210cd192855-kube-api-access-9r4xj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:41.100706 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.100671 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:41.100902 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.100805 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:41.101043 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.101023 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:41.101108 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.101086 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:41.130122 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.130104 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:41.141543 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.141517 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4xj\" (UniqueName: \"kubernetes.io/projected/0495b7c6-52cf-411c-bc40-d210cd192855-kube-api-access-9r4xj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:41.226921 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.226892 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:41.268399 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.268363 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t"] Oct 29 22:45:41.276617 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.274478 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.293123 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.293095 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 22:45:41.293241 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.293124 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 22:45:41.293241 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.293108 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 22:45:41.293241 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.293181 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 22:45:41.293241 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.293190 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:45:41.293241 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.293228 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:45:41.293440 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.293272 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7dsjj\"" Oct 29 22:45:41.296932 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.296911 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t"] Oct 29 22:45:41.302570 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.302549 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.302682 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.302600 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.302773 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.302755 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw28s\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-kube-api-access-xw28s\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.302811 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.302785 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.302852 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.302828 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.302918 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.302900 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.302969 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.302940 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.404182 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.404119 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.404182 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.404155 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.404395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.404203 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw28s\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-kube-api-access-xw28s\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.404395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.404229 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.404395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.404268 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.404395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.404298 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.404395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.404340 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.404885 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.404861 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.406602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.406562 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.406688 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.406626 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.406918 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.406901 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.406985 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.406936 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.424139 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.424120 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs"] Oct 29 22:45:41.426135 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.426115 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.426761 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.426742 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw28s\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-kube-api-access-xw28s\") pod \"istiod-openshift-gateway-7876bfcb74-tmj5t\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.429273 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:45:41.429242 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24907517_a224_4ef7_910d_e975d9f2c517.slice/crio-e8e2acbb21cc79e1ae841f7594b250d532ca8c2db84119c5817702fba71ff5fe WatchSource:0}: Error finding container e8e2acbb21cc79e1ae841f7594b250d532ca8c2db84119c5817702fba71ff5fe: Status 404 returned error can't find the container with id e8e2acbb21cc79e1ae841f7594b250d532ca8c2db84119c5817702fba71ff5fe Oct 29 22:45:41.431194 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.431180 5736 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:45:41.520125 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.520100 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8"] Oct 29 22:45:41.524932 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:45:41.524906 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0495b7c6_52cf_411c_bc40_d210cd192855.slice/crio-44a82f7b4cb6540a983368e64b3a256e81d2faf09db0ea7de644b52388d6a8b0 WatchSource:0}: Error finding container 44a82f7b4cb6540a983368e64b3a256e81d2faf09db0ea7de644b52388d6a8b0: Status 404 returned error can't find the container with id 44a82f7b4cb6540a983368e64b3a256e81d2faf09db0ea7de644b52388d6a8b0 Oct 29 22:45:41.536200 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.536173 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" event={"ID":"24907517-a224-4ef7-910d-e975d9f2c517","Type":"ContainerStarted","Data":"e8e2acbb21cc79e1ae841f7594b250d532ca8c2db84119c5817702fba71ff5fe"} Oct 29 22:45:41.537286 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.537265 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" event={"ID":"0495b7c6-52cf-411c-bc40-d210cd192855","Type":"ContainerStarted","Data":"44a82f7b4cb6540a983368e64b3a256e81d2faf09db0ea7de644b52388d6a8b0"} Oct 29 22:45:41.591568 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.591544 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:41.902710 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:41.902635 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t"] Oct 29 22:45:41.907916 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:45:41.907885 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ccb3dc_63dc_4317_94a1_5f7ea80ae2b1.slice/crio-ba02ee8e10e5ed79e7badcb263144bef6a1281674fc31d8f6fa54a93dfd03480 WatchSource:0}: Error finding container ba02ee8e10e5ed79e7badcb263144bef6a1281674fc31d8f6fa54a93dfd03480: Status 404 returned error can't find the container with id ba02ee8e10e5ed79e7badcb263144bef6a1281674fc31d8f6fa54a93dfd03480 Oct 29 22:45:42.543021 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:42.542987 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" event={"ID":"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1","Type":"ContainerStarted","Data":"ba02ee8e10e5ed79e7badcb263144bef6a1281674fc31d8f6fa54a93dfd03480"} Oct 29 22:45:45.331008 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:45.330967 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:45.331321 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:45.331032 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:45.559002 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:45.558966 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" event={"ID":"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1","Type":"ContainerStarted","Data":"66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c"} Oct 29 22:45:45.559185 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:45.559114 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:45.600703 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:45.600599 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" podStartSLOduration=1.180809254 podStartE2EDuration="4.600559882s" totalImagesPullingTime="3.419750628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:45.600069427 +0000 UTC m=+142.715989036" watchObservedRunningTime="2026-10-29 22:45:45.600559882 +0000 UTC m=+142.716479490" Oct 29 22:45:46.566151 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:46.566120 5736 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-tmj5t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:45:46.566711 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:46.566174 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" podUID="a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:45:48.576517 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:48.576478 5736 generic.go:356] "Generic (PLEG): container finished" podID="24907517-a224-4ef7-910d-e975d9f2c517" containerID="0ef847ac7bda96fcf7e7807fdc316485bcc8d76806014dbd617c6ea7357f1413" exitCode=0 Oct 29 22:45:48.577016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:48.576565 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" event={"ID":"24907517-a224-4ef7-910d-e975d9f2c517","Type":"ContainerDied","Data":"0ef847ac7bda96fcf7e7807fdc316485bcc8d76806014dbd617c6ea7357f1413"} Oct 29 22:45:48.578193 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:48.578170 5736 generic.go:356] "Generic (PLEG): container finished" podID="0495b7c6-52cf-411c-bc40-d210cd192855" containerID="1cab7d8114ad672b6871448e1628f53dda47ce31e1197167e80b1b66a4ec2e78" exitCode=0 Oct 29 22:45:48.578288 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:48.578216 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" event={"ID":"0495b7c6-52cf-411c-bc40-d210cd192855","Type":"ContainerDied","Data":"1cab7d8114ad672b6871448e1628f53dda47ce31e1197167e80b1b66a4ec2e78"} Oct 29 22:45:49.565284 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:49.565253 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:45:50.589121 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:50.589093 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" event={"ID":"0495b7c6-52cf-411c-bc40-d210cd192855","Type":"ContainerStarted","Data":"6cad4df2842dc12db14675635bc54085d0889ad232c7691c7fd01ffb77f82a70"} Oct 29 22:45:51.594320 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:51.594249 5736 generic.go:356] "Generic (PLEG): container finished" podID="24907517-a224-4ef7-910d-e975d9f2c517" containerID="884dd662cbb44c4e5abe225eb986a526f7868e5b948aa7be06dcde1ff9992adb" exitCode=0 Oct 29 22:45:51.594766 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:51.594343 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" event={"ID":"24907517-a224-4ef7-910d-e975d9f2c517","Type":"ContainerDied","Data":"884dd662cbb44c4e5abe225eb986a526f7868e5b948aa7be06dcde1ff9992adb"} Oct 29 22:45:51.596364 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:51.596256 5736 generic.go:356] "Generic (PLEG): container finished" podID="0495b7c6-52cf-411c-bc40-d210cd192855" containerID="6cad4df2842dc12db14675635bc54085d0889ad232c7691c7fd01ffb77f82a70" exitCode=0 Oct 29 22:45:51.596364 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:51.596281 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" event={"ID":"0495b7c6-52cf-411c-bc40-d210cd192855","Type":"ContainerDied","Data":"6cad4df2842dc12db14675635bc54085d0889ad232c7691c7fd01ffb77f82a70"} Oct 29 22:45:56.829733 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:45:56.829689 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:45:58.634017 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:58.633980 5736 generic.go:356] "Generic (PLEG): container finished" podID="24907517-a224-4ef7-910d-e975d9f2c517" containerID="312d8e21e9c16de8df425e1c00f13cc02221f79c46a0bf9aabce046fbf7e82ca" exitCode=0 Oct 29 22:45:58.634493 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:58.634058 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" event={"ID":"24907517-a224-4ef7-910d-e975d9f2c517","Type":"ContainerDied","Data":"312d8e21e9c16de8df425e1c00f13cc02221f79c46a0bf9aabce046fbf7e82ca"} Oct 29 22:45:58.636197 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:58.636176 5736 generic.go:356] "Generic (PLEG): container finished" podID="0495b7c6-52cf-411c-bc40-d210cd192855" containerID="71c06be107b02eaca5ee7993b0be84d143b6fc8b65c78109957a0de57263425b" exitCode=0 Oct 29 22:45:58.636310 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:58.636217 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" event={"ID":"0495b7c6-52cf-411c-bc40-d210cd192855","Type":"ContainerDied","Data":"71c06be107b02eaca5ee7993b0be84d143b6fc8b65c78109957a0de57263425b"} Oct 29 22:45:59.771269 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.771243 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:45:59.793429 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.793405 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:45:59.878387 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.878356 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-bundle\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-bundle\") pod \"24907517-a224-4ef7-910d-e975d9f2c517\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " Oct 29 22:45:59.878558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.878395 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24907517-a224-4ef7-910d-e975d9f2c517-kube-api-access-f4rd5\" (UniqueName: \"kubernetes.io/projected/24907517-a224-4ef7-910d-e975d9f2c517-kube-api-access-f4rd5\") pod \"24907517-a224-4ef7-910d-e975d9f2c517\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " Oct 29 22:45:59.878558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.878450 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0495b7c6-52cf-411c-bc40-d210cd192855-kube-api-access-9r4xj\" (UniqueName: \"kubernetes.io/projected/0495b7c6-52cf-411c-bc40-d210cd192855-kube-api-access-9r4xj\") pod \"0495b7c6-52cf-411c-bc40-d210cd192855\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " Oct 29 22:45:59.878558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.878545 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-bundle\") pod \"0495b7c6-52cf-411c-bc40-d210cd192855\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " Oct 29 22:45:59.878754 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.878597 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-util\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-util\") pod \"0495b7c6-52cf-411c-bc40-d210cd192855\" (UID: \"0495b7c6-52cf-411c-bc40-d210cd192855\") " Oct 29 22:45:59.878754 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.878634 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-util\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-util\") pod \"24907517-a224-4ef7-910d-e975d9f2c517\" (UID: \"24907517-a224-4ef7-910d-e975d9f2c517\") " Oct 29 22:45:59.878970 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.878942 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-bundle" pod "24907517-a224-4ef7-910d-e975d9f2c517" (UID: "24907517-a224-4ef7-910d-e975d9f2c517"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:59.879407 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.879379 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-bundle" pod "0495b7c6-52cf-411c-bc40-d210cd192855" (UID: "0495b7c6-52cf-411c-bc40-d210cd192855"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:59.880777 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.880747 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24907517-a224-4ef7-910d-e975d9f2c517-kube-api-access-f4rd5" pod "24907517-a224-4ef7-910d-e975d9f2c517" (UID: "24907517-a224-4ef7-910d-e975d9f2c517"). InnerVolumeSpecName "kube-api-access-f4rd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:59.881032 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.881014 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0495b7c6-52cf-411c-bc40-d210cd192855-kube-api-access-9r4xj" pod "0495b7c6-52cf-411c-bc40-d210cd192855" (UID: "0495b7c6-52cf-411c-bc40-d210cd192855"). InnerVolumeSpecName "kube-api-access-9r4xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:59.884316 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.884293 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-util" pod "0495b7c6-52cf-411c-bc40-d210cd192855" (UID: "0495b7c6-52cf-411c-bc40-d210cd192855"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:59.885045 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.885013 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-util" pod "24907517-a224-4ef7-910d-e975d9f2c517" (UID: "24907517-a224-4ef7-910d-e975d9f2c517"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:59.979702 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.979670 5736 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-util\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:45:59.979845 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.979720 5736 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24907517-a224-4ef7-910d-e975d9f2c517-bundle\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:45:59.979845 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.979731 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4rd5\" (UniqueName: \"kubernetes.io/projected/24907517-a224-4ef7-910d-e975d9f2c517-kube-api-access-f4rd5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:45:59.979845 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.979742 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r4xj\" (UniqueName: \"kubernetes.io/projected/0495b7c6-52cf-411c-bc40-d210cd192855-kube-api-access-9r4xj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:45:59.979845 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.979750 5736 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-bundle\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:45:59.979845 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:45:59.979758 5736 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0495b7c6-52cf-411c-bc40-d210cd192855-util\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:46:00.645594 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:00.645549 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" Oct 29 22:46:00.645795 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:00.645549 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445glbs" event={"ID":"24907517-a224-4ef7-910d-e975d9f2c517","Type":"ContainerDied","Data":"e8e2acbb21cc79e1ae841f7594b250d532ca8c2db84119c5817702fba71ff5fe"} Oct 29 22:46:00.645795 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:00.645676 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8e2acbb21cc79e1ae841f7594b250d532ca8c2db84119c5817702fba71ff5fe" Oct 29 22:46:00.647431 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:00.647409 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" Oct 29 22:46:00.647563 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:00.647406 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c84rp8" event={"ID":"0495b7c6-52cf-411c-bc40-d210cd192855","Type":"ContainerDied","Data":"44a82f7b4cb6540a983368e64b3a256e81d2faf09db0ea7de644b52388d6a8b0"} Oct 29 22:46:00.647563 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:00.647523 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44a82f7b4cb6540a983368e64b3a256e81d2faf09db0ea7de644b52388d6a8b0" Oct 29 22:46:26.822045 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:46:26.822005 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:46:28.556957 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.556918 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4"] Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557248 5736 cpu_manager.go:414] "containerMap: removing container" podUID="0495b7c6-52cf-411c-bc40-d210cd192855" containerName="util" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557260 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0495b7c6-52cf-411c-bc40-d210cd192855" containerName="util" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557283 5736 cpu_manager.go:414] "containerMap: removing container" podUID="24907517-a224-4ef7-910d-e975d9f2c517" containerName="util" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557288 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24907517-a224-4ef7-910d-e975d9f2c517" containerName="util" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557294 5736 cpu_manager.go:414] "containerMap: removing container" podUID="0495b7c6-52cf-411c-bc40-d210cd192855" containerName="extract" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557300 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0495b7c6-52cf-411c-bc40-d210cd192855" containerName="extract" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557308 5736 cpu_manager.go:414] "containerMap: removing container" podUID="24907517-a224-4ef7-910d-e975d9f2c517" containerName="pull" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557312 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24907517-a224-4ef7-910d-e975d9f2c517" containerName="pull" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557323 5736 cpu_manager.go:414] "containerMap: removing container" podUID="24907517-a224-4ef7-910d-e975d9f2c517" containerName="extract" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557327 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24907517-a224-4ef7-910d-e975d9f2c517" containerName="extract" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557335 5736 cpu_manager.go:414] "containerMap: removing container" podUID="0495b7c6-52cf-411c-bc40-d210cd192855" containerName="pull" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557339 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0495b7c6-52cf-411c-bc40-d210cd192855" containerName="pull" Oct 29 22:46:28.557384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557391 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="0495b7c6-52cf-411c-bc40-d210cd192855" containerName="extract" Oct 29 22:46:28.557879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.557399 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="24907517-a224-4ef7-910d-e975d9f2c517" containerName="extract" Oct 29 22:46:28.560815 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.560799 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.611424 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.611396 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rbf\" (UniqueName: \"kubernetes.io/projected/52857f34-4eba-4768-b30b-0496765a631e-kube-api-access-75rbf\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.611567 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.611433 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52857f34-4eba-4768-b30b-0496765a631e-metrics-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.611567 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.611454 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52857f34-4eba-4768-b30b-0496765a631e-tmp\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.611707 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.611560 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52857f34-4eba-4768-b30b-0496765a631e-webhook-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.611707 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.611625 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52857f34-4eba-4768-b30b-0496765a631e-manifests\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.611707 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.611658 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52857f34-4eba-4768-b30b-0496765a631e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.654620 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.654566 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 22:46:28.654620 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.654566 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-c59b2\"" Oct 29 22:46:28.654800 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.654679 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 22:46:28.654800 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.654683 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 29 22:46:28.656977 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.656943 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4"] Oct 29 22:46:28.665925 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.665903 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 29 22:46:28.666037 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.665959 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 29 22:46:28.712439 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.712403 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52857f34-4eba-4768-b30b-0496765a631e-webhook-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.712610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.712449 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52857f34-4eba-4768-b30b-0496765a631e-manifests\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.712610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.712487 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52857f34-4eba-4768-b30b-0496765a631e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.712610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.712514 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rbf\" (UniqueName: \"kubernetes.io/projected/52857f34-4eba-4768-b30b-0496765a631e-kube-api-access-75rbf\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.712610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.712533 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52857f34-4eba-4768-b30b-0496765a631e-metrics-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.712824 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.712697 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52857f34-4eba-4768-b30b-0496765a631e-tmp\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.712959 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.712940 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52857f34-4eba-4768-b30b-0496765a631e-manifests\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.714860 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.714834 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52857f34-4eba-4768-b30b-0496765a631e-tmp\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.715055 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.715032 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52857f34-4eba-4768-b30b-0496765a631e-webhook-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.715135 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.715117 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52857f34-4eba-4768-b30b-0496765a631e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.715195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.715129 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52857f34-4eba-4768-b30b-0496765a631e-metrics-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.743892 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.743866 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rbf\" (UniqueName: \"kubernetes.io/projected/52857f34-4eba-4768-b30b-0496765a631e-kube-api-access-75rbf\") pod \"opendatahub-operator-controller-manager-844ffc67f7-55th4\" (UID: \"52857f34-4eba-4768-b30b-0496765a631e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:28.873568 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:28.873494 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:29.216851 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:29.216816 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4"] Oct 29 22:46:29.220778 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:46:29.220753 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52857f34_4eba_4768_b30b_0496765a631e.slice/crio-0e301489dea874bd19df5a7534c1b91b4e4e2aadafcb2f0036ac6cbec9fbd062 WatchSource:0}: Error finding container 0e301489dea874bd19df5a7534c1b91b4e4e2aadafcb2f0036ac6cbec9fbd062: Status 404 returned error can't find the container with id 0e301489dea874bd19df5a7534c1b91b4e4e2aadafcb2f0036ac6cbec9fbd062 Oct 29 22:46:29.757956 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:29.757920 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" event={"ID":"52857f34-4eba-4768-b30b-0496765a631e","Type":"ContainerStarted","Data":"0e301489dea874bd19df5a7534c1b91b4e4e2aadafcb2f0036ac6cbec9fbd062"} Oct 29 22:46:32.778595 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:32.778556 5736 generic.go:356] "Generic (PLEG): container finished" podID="52857f34-4eba-4768-b30b-0496765a631e" containerID="7bdbd4ede38c5473b605917462cc701eabd91febac34e89bec3b1878d1608f16" exitCode=0 Oct 29 22:46:32.779049 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:32.778642 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" event={"ID":"52857f34-4eba-4768-b30b-0496765a631e","Type":"ContainerDied","Data":"7bdbd4ede38c5473b605917462cc701eabd91febac34e89bec3b1878d1608f16"} Oct 29 22:46:33.784163 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:33.784131 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" event={"ID":"52857f34-4eba-4768-b30b-0496765a631e","Type":"ContainerStarted","Data":"c4415847fe8b4f0d522c12a4abef4cb2d4b9d07b9cbf8252cedcc27aeed4ab79"} Oct 29 22:46:33.784543 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:33.784262 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:33.889412 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:33.889309 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" podStartSLOduration=2.334809914 podStartE2EDuration="5.889294572s" totalImagesPullingTime="3.131849355s" totalInitContainerRuntime="422.635303ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:33.878326448 +0000 UTC m=+190.994246055" watchObservedRunningTime="2026-10-29 22:46:33.889294572 +0000 UTC m=+191.005214179" Oct 29 22:46:44.790764 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:46:44.790734 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-55th4" Oct 29 22:46:56.822366 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:46:56.822331 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:47:02.956364 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:02.956334 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67f887f896-8g7tq"] Oct 29 22:47:02.964781 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:02.964755 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:02.984575 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:02.984543 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67f887f896-8g7tq"] Oct 29 22:47:02.994431 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:02.994397 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 29 22:47:02.994431 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:02.994426 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 29 22:47:02.994640 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:02.994547 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6mfk4\"" Oct 29 22:47:03.096753 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.096718 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/323481e4-8d92-49db-aef1-5ac5b469271e-tls-certs\") pod \"kube-auth-proxy-67f887f896-8g7tq\" (UID: \"323481e4-8d92-49db-aef1-5ac5b469271e\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.096912 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.096774 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5r9f\" (UniqueName: \"kubernetes.io/projected/323481e4-8d92-49db-aef1-5ac5b469271e-kube-api-access-d5r9f\") pod \"kube-auth-proxy-67f887f896-8g7tq\" (UID: \"323481e4-8d92-49db-aef1-5ac5b469271e\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.096992 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.096916 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/323481e4-8d92-49db-aef1-5ac5b469271e-tmp\") pod \"kube-auth-proxy-67f887f896-8g7tq\" (UID: \"323481e4-8d92-49db-aef1-5ac5b469271e\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.197343 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.197312 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/323481e4-8d92-49db-aef1-5ac5b469271e-tls-certs\") pod \"kube-auth-proxy-67f887f896-8g7tq\" (UID: \"323481e4-8d92-49db-aef1-5ac5b469271e\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.197522 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.197358 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5r9f\" (UniqueName: \"kubernetes.io/projected/323481e4-8d92-49db-aef1-5ac5b469271e-kube-api-access-d5r9f\") pod \"kube-auth-proxy-67f887f896-8g7tq\" (UID: \"323481e4-8d92-49db-aef1-5ac5b469271e\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.197522 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.197407 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/323481e4-8d92-49db-aef1-5ac5b469271e-tmp\") pod \"kube-auth-proxy-67f887f896-8g7tq\" (UID: \"323481e4-8d92-49db-aef1-5ac5b469271e\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.199894 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.199841 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/323481e4-8d92-49db-aef1-5ac5b469271e-tmp\") pod \"kube-auth-proxy-67f887f896-8g7tq\" (UID: \"323481e4-8d92-49db-aef1-5ac5b469271e\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.200270 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.200245 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/323481e4-8d92-49db-aef1-5ac5b469271e-tls-certs\") pod \"kube-auth-proxy-67f887f896-8g7tq\" (UID: \"323481e4-8d92-49db-aef1-5ac5b469271e\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.219120 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.219098 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5r9f\" (UniqueName: \"kubernetes.io/projected/323481e4-8d92-49db-aef1-5ac5b469271e-kube-api-access-d5r9f\") pod \"kube-auth-proxy-67f887f896-8g7tq\" (UID: \"323481e4-8d92-49db-aef1-5ac5b469271e\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.281266 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.281228 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" Oct 29 22:47:03.604960 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.604869 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67f887f896-8g7tq"] Oct 29 22:47:03.608910 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:47:03.608871 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323481e4_8d92_49db_aef1_5ac5b469271e.slice/crio-b33c29222ecd286c050db693985dd0dc1a7a191b3f889e5fb85617cdcb3cb608 WatchSource:0}: Error finding container b33c29222ecd286c050db693985dd0dc1a7a191b3f889e5fb85617cdcb3cb608: Status 404 returned error can't find the container with id b33c29222ecd286c050db693985dd0dc1a7a191b3f889e5fb85617cdcb3cb608 Oct 29 22:47:03.909154 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:03.909065 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" event={"ID":"323481e4-8d92-49db-aef1-5ac5b469271e","Type":"ContainerStarted","Data":"b33c29222ecd286c050db693985dd0dc1a7a191b3f889e5fb85617cdcb3cb608"} Oct 29 22:47:06.923888 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:06.923843 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" event={"ID":"323481e4-8d92-49db-aef1-5ac5b469271e","Type":"ContainerStarted","Data":"332a6700671923761964ff4678f8a0f05a9ba38464c2515c514e6deeb30e52ce"} Oct 29 22:47:06.965313 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:06.965253 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67f887f896-8g7tq" podStartSLOduration=2.624372902 podStartE2EDuration="4.965236072s" totalImagesPullingTime="2.34086317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:06.964518111 +0000 UTC m=+224.080437717" watchObservedRunningTime="2026-10-29 22:47:06.965236072 +0000 UTC m=+224.081155677" Oct 29 22:47:17.396672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.396635 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z975q"] Oct 29 22:47:17.398765 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.398743 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:17.412462 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.412439 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 29 22:47:17.423072 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.423049 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 22:47:17.423210 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.423191 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 22:47:17.423277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.423202 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8kwg7\"" Oct 29 22:47:17.423652 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.423624 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-config-file\") pod \"limitador-limitador-74dfcfb894-z975q\" (UID: \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:17.423777 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.423668 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlgm5\" (UniqueName: \"kubernetes.io/projected/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-kube-api-access-jlgm5\") pod \"limitador-limitador-74dfcfb894-z975q\" (UID: \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:17.424427 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.424407 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z975q"] Oct 29 22:47:17.524211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.524182 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-config-file\") pod \"limitador-limitador-74dfcfb894-z975q\" (UID: \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:17.524385 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.524217 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlgm5\" (UniqueName: \"kubernetes.io/projected/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-kube-api-access-jlgm5\") pod \"limitador-limitador-74dfcfb894-z975q\" (UID: \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:17.524802 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.524783 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-config-file\") pod \"limitador-limitador-74dfcfb894-z975q\" (UID: \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:17.554244 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.552699 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z975q"] Oct 29 22:47:17.560471 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.560446 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlgm5\" (UniqueName: \"kubernetes.io/projected/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-kube-api-access-jlgm5\") pod \"limitador-limitador-74dfcfb894-z975q\" (UID: \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:17.710498 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:17.710476 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:18.009471 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:18.009395 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z975q"] Oct 29 22:47:18.014474 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:47:18.014446 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1818830e_b6c1_4ff8_ba4b_79289e0a8ef5.slice/crio-b7cf2c0d98d68f508c4a0264fe3d31d876e8e16a0abcb10c983b11a7b81f093d WatchSource:0}: Error finding container b7cf2c0d98d68f508c4a0264fe3d31d876e8e16a0abcb10c983b11a7b81f093d: Status 404 returned error can't find the container with id b7cf2c0d98d68f508c4a0264fe3d31d876e8e16a0abcb10c983b11a7b81f093d Oct 29 22:47:18.979922 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:18.979873 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" event={"ID":"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5","Type":"ContainerStarted","Data":"b7cf2c0d98d68f508c4a0264fe3d31d876e8e16a0abcb10c983b11a7b81f093d"} Oct 29 22:47:20.989659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:20.989624 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" event={"ID":"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5","Type":"ContainerStarted","Data":"1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7"} Oct 29 22:47:20.990054 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:20.989708 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:21.029982 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:21.029927 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" podStartSLOduration=2.030725393 podStartE2EDuration="4.029909157s" totalImagesPullingTime="1.999183764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:21.02953648 +0000 UTC m=+238.145456083" watchObservedRunningTime="2026-10-29 22:47:21.029909157 +0000 UTC m=+238.145828765" Oct 29 22:47:26.822212 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:47:26.822174 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:47:30.925703 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:30.925667 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z975q"] Oct 29 22:47:30.926312 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:30.926009 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" podUID="1818830e-b6c1-4ff8-ba4b-79289e0a8ef5" containerName="limitador" containerID="cri-o://1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7" gracePeriod=30 Oct 29 22:47:30.926692 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:30.926667 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:31.485735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:31.485680 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:31.535204 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:31.535173 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-config-file\" (UniqueName: \"kubernetes.io/configmap/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-config-file\") pod \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\" (UID: \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\") " Oct 29 22:47:31.535377 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:31.535261 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-kube-api-access-jlgm5\" (UniqueName: \"kubernetes.io/projected/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-kube-api-access-jlgm5\") pod \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\" (UID: \"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5\") " Oct 29 22:47:31.535665 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:31.535632 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-config-file" pod "1818830e-b6c1-4ff8-ba4b-79289e0a8ef5" (UID: "1818830e-b6c1-4ff8-ba4b-79289e0a8ef5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:31.537510 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:31.537481 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-kube-api-access-jlgm5" pod "1818830e-b6c1-4ff8-ba4b-79289e0a8ef5" (UID: "1818830e-b6c1-4ff8-ba4b-79289e0a8ef5"). InnerVolumeSpecName "kube-api-access-jlgm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:31.636657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:31.636621 5736 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-config-file\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:47:31.636657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:31.636649 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlgm5\" (UniqueName: \"kubernetes.io/projected/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5-kube-api-access-jlgm5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:47:32.034371 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:32.034336 5736 generic.go:356] "Generic (PLEG): container finished" podID="1818830e-b6c1-4ff8-ba4b-79289e0a8ef5" containerID="1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7" exitCode=0 Oct 29 22:47:32.034791 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:32.034383 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" Oct 29 22:47:32.034791 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:32.034423 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" event={"ID":"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5","Type":"ContainerDied","Data":"1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7"} Oct 29 22:47:32.034791 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:32.034476 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z975q" event={"ID":"1818830e-b6c1-4ff8-ba4b-79289e0a8ef5","Type":"ContainerDied","Data":"b7cf2c0d98d68f508c4a0264fe3d31d876e8e16a0abcb10c983b11a7b81f093d"} Oct 29 22:47:32.034791 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:32.034497 5736 scope.go:122] "RemoveContainer" containerID="1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7" Oct 29 22:47:32.043060 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:32.043043 5736 scope.go:122] "RemoveContainer" containerID="1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7" Oct 29 22:47:32.043366 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:47:32.043343 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7\": container with ID starting with 1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7 not found: ID does not exist" containerID="1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7" Oct 29 22:47:32.043434 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:32.043374 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7"} err="failed to get container status \"1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7\": rpc error: code = NotFound desc = could not find container \"1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7\": container with ID starting with 1039bff4f4e7ce8bff9a31380ef134ac8e577099aefd902cd2f1487e8a6ff6a7 not found: ID does not exist" Oct 29 22:47:32.079413 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:32.079390 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z975q"] Oct 29 22:47:32.090001 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:32.089978 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z975q"] Oct 29 22:47:33.002637 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:33.002603 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1818830e-b6c1-4ff8-ba4b-79289e0a8ef5" path="/var/lib/kubelet/pods/1818830e-b6c1-4ff8-ba4b-79289e0a8ef5/volumes" Oct 29 22:47:35.824225 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.824194 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h"] Oct 29 22:47:35.824674 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.824594 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1818830e-b6c1-4ff8-ba4b-79289e0a8ef5" containerName="limitador" Oct 29 22:47:35.824674 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.824617 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1818830e-b6c1-4ff8-ba4b-79289e0a8ef5" containerName="limitador" Oct 29 22:47:35.824793 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.824693 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1818830e-b6c1-4ff8-ba4b-79289e0a8ef5" containerName="limitador" Oct 29 22:47:35.827627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.827605 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:35.851617 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.851227 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-lp5cl\"" Oct 29 22:47:35.851617 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.851232 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 29 22:47:35.851617 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.851301 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:47:35.851617 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.851330 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:47:35.852078 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.852057 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h"] Oct 29 22:47:35.871254 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.871229 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9f3a369-0f84-4d09-bd71-180facff85a0-data\") pod \"postgres-7cdbf6dc44-x4z7h\" (UID: \"c9f3a369-0f84-4d09-bd71-180facff85a0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:35.871428 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.871412 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz4pg\" (UniqueName: \"kubernetes.io/projected/c9f3a369-0f84-4d09-bd71-180facff85a0-kube-api-access-sz4pg\") pod \"postgres-7cdbf6dc44-x4z7h\" (UID: \"c9f3a369-0f84-4d09-bd71-180facff85a0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:35.972830 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.972796 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz4pg\" (UniqueName: \"kubernetes.io/projected/c9f3a369-0f84-4d09-bd71-180facff85a0-kube-api-access-sz4pg\") pod \"postgres-7cdbf6dc44-x4z7h\" (UID: \"c9f3a369-0f84-4d09-bd71-180facff85a0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:35.973014 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.972865 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9f3a369-0f84-4d09-bd71-180facff85a0-data\") pod \"postgres-7cdbf6dc44-x4z7h\" (UID: \"c9f3a369-0f84-4d09-bd71-180facff85a0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:35.973261 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.973240 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9f3a369-0f84-4d09-bd71-180facff85a0-data\") pod \"postgres-7cdbf6dc44-x4z7h\" (UID: \"c9f3a369-0f84-4d09-bd71-180facff85a0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:35.994634 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:35.994610 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz4pg\" (UniqueName: \"kubernetes.io/projected/c9f3a369-0f84-4d09-bd71-180facff85a0-kube-api-access-sz4pg\") pod \"postgres-7cdbf6dc44-x4z7h\" (UID: \"c9f3a369-0f84-4d09-bd71-180facff85a0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:36.140380 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:36.140302 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:36.433633 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:36.433501 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h"] Oct 29 22:47:36.438590 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:47:36.438545 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f3a369_0f84_4d09_bd71_180facff85a0.slice/crio-86876b6b97520cb3950622fb9731f6f390ec0554423ed89e197ea472a047a644 WatchSource:0}: Error finding container 86876b6b97520cb3950622fb9731f6f390ec0554423ed89e197ea472a047a644: Status 404 returned error can't find the container with id 86876b6b97520cb3950622fb9731f6f390ec0554423ed89e197ea472a047a644 Oct 29 22:47:37.058589 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:37.058556 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" event={"ID":"c9f3a369-0f84-4d09-bd71-180facff85a0","Type":"ContainerStarted","Data":"86876b6b97520cb3950622fb9731f6f390ec0554423ed89e197ea472a047a644"} Oct 29 22:47:41.226151 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.226121 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c"] Oct 29 22:47:41.232960 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.232936 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.249768 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.249743 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 29 22:47:41.260632 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.260611 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 29 22:47:41.260632 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.260613 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:47:41.260804 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.260631 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 29 22:47:41.261494 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.261471 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c"] Oct 29 22:47:41.325254 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.325228 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12e8bdbf-3162-4499-9066-fba51014d23a-manager-config\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.325353 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.325310 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12e8bdbf-3162-4499-9066-fba51014d23a-cert\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.325435 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.325415 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4tf\" (UniqueName: \"kubernetes.io/projected/12e8bdbf-3162-4499-9066-fba51014d23a-kube-api-access-cs4tf\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.325495 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.325450 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e8bdbf-3162-4499-9066-fba51014d23a-metrics-certs\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.351641 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.351624 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-9cznk\"" Oct 29 22:47:41.351733 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.351671 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 29 22:47:41.426169 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.426136 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12e8bdbf-3162-4499-9066-fba51014d23a-manager-config\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.426313 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.426191 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12e8bdbf-3162-4499-9066-fba51014d23a-cert\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.426313 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.426235 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4tf\" (UniqueName: \"kubernetes.io/projected/12e8bdbf-3162-4499-9066-fba51014d23a-kube-api-access-cs4tf\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.426313 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.426252 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e8bdbf-3162-4499-9066-fba51014d23a-metrics-certs\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.426766 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.426744 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12e8bdbf-3162-4499-9066-fba51014d23a-manager-config\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.428617 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.428595 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e8bdbf-3162-4499-9066-fba51014d23a-metrics-certs\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.428709 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.428644 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12e8bdbf-3162-4499-9066-fba51014d23a-cert\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.447744 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.447720 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4tf\" (UniqueName: \"kubernetes.io/projected/12e8bdbf-3162-4499-9066-fba51014d23a-kube-api-access-cs4tf\") pod \"jobset-controller-manager-559fc64dd7-m964c\" (UID: \"12e8bdbf-3162-4499-9066-fba51014d23a\") " pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.544822 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.544765 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:41.842093 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:41.841990 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c"] Oct 29 22:47:41.847311 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:47:41.847281 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e8bdbf_3162_4499_9066_fba51014d23a.slice/crio-ee6819e884ee731ac5ad47017dfebb29938e2996a8e30491a2f72d277a39c47b WatchSource:0}: Error finding container ee6819e884ee731ac5ad47017dfebb29938e2996a8e30491a2f72d277a39c47b: Status 404 returned error can't find the container with id ee6819e884ee731ac5ad47017dfebb29938e2996a8e30491a2f72d277a39c47b Oct 29 22:47:42.084184 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:42.084146 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" event={"ID":"c9f3a369-0f84-4d09-bd71-180facff85a0","Type":"ContainerStarted","Data":"e07fb44ea98cdc1ba634075fb8e98092506a6a1504ecd9a1b093d7bc0ca51158"} Oct 29 22:47:42.084352 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:42.084231 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:42.085448 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:42.085429 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" event={"ID":"12e8bdbf-3162-4499-9066-fba51014d23a","Type":"ContainerStarted","Data":"ee6819e884ee731ac5ad47017dfebb29938e2996a8e30491a2f72d277a39c47b"} Oct 29 22:47:42.121018 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:42.120948 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" podStartSLOduration=2.250806769 podStartE2EDuration="7.120936267s" totalImagesPullingTime="4.870129498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:42.120148614 +0000 UTC m=+259.236068219" watchObservedRunningTime="2026-10-29 22:47:42.120936267 +0000 UTC m=+259.236855928" Oct 29 22:47:45.102673 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:45.102631 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" event={"ID":"12e8bdbf-3162-4499-9066-fba51014d23a","Type":"ContainerStarted","Data":"2c377259e81a5f959605a3d4a525ea2dfafff1bba5731272ec03282306140a7b"} Oct 29 22:47:45.103179 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:45.102762 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:45.140649 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:45.140600 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" podStartSLOduration=1.708407558 podStartE2EDuration="4.140567679s" totalImagesPullingTime="2.432160121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:45.14004517 +0000 UTC m=+262.255964776" watchObservedRunningTime="2026-10-29 22:47:45.140567679 +0000 UTC m=+262.256487263" Oct 29 22:47:48.120458 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:48.120430 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x4z7h" Oct 29 22:47:56.113401 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:47:56.113370 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-559fc64dd7-m964c" Oct 29 22:47:56.821937 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:47:56.821895 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:48:10.223552 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.223339 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t"] Oct 29 22:48:10.224159 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.223756 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" podUID="a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" containerName="discovery" containerID="cri-o://66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c" gracePeriod=30 Oct 29 22:48:10.463915 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.463893 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:48:10.472413 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.472389 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-local-certs\") pod \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " Oct 29 22:48:10.472528 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.472448 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-kubeconfig\") pod \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " Oct 29 22:48:10.472528 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.472490 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-kube-api-access-xw28s\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-kube-api-access-xw28s\") pod \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " Oct 29 22:48:10.472669 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.472557 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-dns-cert\") pod \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " Oct 29 22:48:10.472669 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.472601 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-ca-configmap\") pod \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " Oct 29 22:48:10.472669 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.472637 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-cacerts\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-cacerts\") pod \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " Oct 29 22:48:10.472669 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.472668 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-token\") pod \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\" (UID: \"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1\") " Oct 29 22:48:10.473321 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.473278 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-ca-configmap" pod "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" (UID: "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:10.475370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.475292 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-local-certs" pod "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" (UID: "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:10.475370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.475307 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-kubeconfig" pod "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" (UID: "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:10.475370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.475323 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-dns-cert" pod "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" (UID: "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:10.475370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.475298 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-token" pod "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" (UID: "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:10.475370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.475296 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-kube-api-access-xw28s" pod "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" (UID: "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1"). InnerVolumeSpecName "kube-api-access-xw28s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:10.475627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.475442 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-cacerts" pod "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" (UID: "a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:10.573254 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.573225 5736 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-cacerts\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.573254 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.573251 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.573254 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.573262 5736 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-local-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.573460 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.573270 5736 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-kubeconfig\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.573460 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.573279 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw28s\" (UniqueName: \"kubernetes.io/projected/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-kube-api-access-xw28s\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.573460 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.573288 5736 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-dns-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:10.573460 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:10.573297 5736 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1-istio-csr-ca-configmap\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:11.221422 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:11.221388 5736 generic.go:356] "Generic (PLEG): container finished" podID="a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" containerID="66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c" exitCode=0 Oct 29 22:48:11.221596 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:11.221454 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" Oct 29 22:48:11.221596 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:11.221474 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" event={"ID":"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1","Type":"ContainerDied","Data":"66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c"} Oct 29 22:48:11.221596 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:11.221532 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t" event={"ID":"a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1","Type":"ContainerDied","Data":"ba02ee8e10e5ed79e7badcb263144bef6a1281674fc31d8f6fa54a93dfd03480"} Oct 29 22:48:11.221596 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:11.221553 5736 scope.go:122] "RemoveContainer" containerID="66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c" Oct 29 22:48:11.230070 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:11.229914 5736 scope.go:122] "RemoveContainer" containerID="66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c" Oct 29 22:48:11.230297 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:48:11.230177 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c\": container with ID starting with 66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c not found: ID does not exist" containerID="66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c" Oct 29 22:48:11.230297 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:11.230208 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c"} err="failed to get container status \"66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c\": rpc error: code = NotFound desc = could not find container \"66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c\": container with ID starting with 66799d60fff5c6e3eb7f9f5485e1c3b8412dc48347b017bbff6faa8641e5a29c not found: ID does not exist" Oct 29 22:48:11.280845 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:11.280439 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t"] Oct 29 22:48:11.284878 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:11.284854 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-tmj5t"] Oct 29 22:48:13.001058 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:13.001018 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" path="/var/lib/kubelet/pods/a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1/volumes" Oct 29 22:48:23.026940 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:23.026916 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:48:23.030016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:23.029988 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:48:26.821813 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:48:26.821771 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:48:31.372360 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.372320 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8f695549b-54g6g"] Oct 29 22:48:31.372995 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.372908 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" containerName="discovery" Oct 29 22:48:31.372995 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.372928 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" containerName="discovery" Oct 29 22:48:31.373103 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.373064 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ccb3dc-63dc-4317-94a1-5f7ea80ae2b1" containerName="discovery" Oct 29 22:48:31.398859 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.398832 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.399649 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.399623 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f695549b-54g6g"] Oct 29 22:48:31.418419 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.418393 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-maas-api-tls\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.418515 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.418431 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-metrics-certs\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.418515 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.418454 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2597b61-cae0-4fb3-a81e-c482e4af83f7-openshift-service-ca\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.418515 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.418508 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m47nq\" (UniqueName: \"kubernetes.io/projected/c2597b61-cae0-4fb3-a81e-c482e4af83f7-kube-api-access-m47nq\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.422749 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.422730 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 29 22:48:31.422821 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.422802 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 22:48:31.422890 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.422878 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pr5wj\"" Oct 29 22:48:31.519349 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.519324 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-maas-api-tls\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.519503 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.519357 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-metrics-certs\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.519503 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.519380 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2597b61-cae0-4fb3-a81e-c482e4af83f7-openshift-service-ca\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.519503 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.519489 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m47nq\" (UniqueName: \"kubernetes.io/projected/c2597b61-cae0-4fb3-a81e-c482e4af83f7-kube-api-access-m47nq\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.520130 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.520109 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2597b61-cae0-4fb3-a81e-c482e4af83f7-openshift-service-ca\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.521739 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.521720 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-metrics-certs\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.521896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.521877 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-maas-api-tls\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.540924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.540895 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m47nq\" (UniqueName: \"kubernetes.io/projected/c2597b61-cae0-4fb3-a81e-c482e4af83f7-kube-api-access-m47nq\") pod \"maas-api-8f695549b-54g6g\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:31.712936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:31.712905 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:32.009318 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:32.009247 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f695549b-54g6g"] Oct 29 22:48:32.014145 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:48:32.014119 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2597b61_cae0_4fb3_a81e_c482e4af83f7.slice/crio-4d996bcc52ec109fbf9b272bc79c379dc65510b25241c8dbd65519cd0b6f3f23 WatchSource:0}: Error finding container 4d996bcc52ec109fbf9b272bc79c379dc65510b25241c8dbd65519cd0b6f3f23: Status 404 returned error can't find the container with id 4d996bcc52ec109fbf9b272bc79c379dc65510b25241c8dbd65519cd0b6f3f23 Oct 29 22:48:32.315061 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:32.314978 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" event={"ID":"c2597b61-cae0-4fb3-a81e-c482e4af83f7","Type":"ContainerStarted","Data":"4d996bcc52ec109fbf9b272bc79c379dc65510b25241c8dbd65519cd0b6f3f23"} Oct 29 22:48:35.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:35.337094 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" event={"ID":"c2597b61-cae0-4fb3-a81e-c482e4af83f7","Type":"ContainerStarted","Data":"24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c"} Oct 29 22:48:35.337526 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:35.337182 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:35.376455 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:35.376405 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" podStartSLOduration=2.117855593 podStartE2EDuration="4.3763893s" totalImagesPullingTime="2.258533707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:35.375237584 +0000 UTC m=+312.491157190" watchObservedRunningTime="2026-10-29 22:48:35.3763893 +0000 UTC m=+312.492308903" Oct 29 22:48:41.346998 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:41.346973 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:42.313482 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.313446 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv"] Oct 29 22:48:42.316979 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.316958 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.343444 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.343414 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv"] Oct 29 22:48:42.410512 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.410484 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1cdd049-d5d5-4a3a-87cb-360491c56536-metrics-certs\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.410879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.410513 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1cdd049-d5d5-4a3a-87cb-360491c56536-maas-api-tls\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.410879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.410544 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1cdd049-d5d5-4a3a-87cb-360491c56536-openshift-service-ca\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.410879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.410709 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7d9k\" (UniqueName: \"kubernetes.io/projected/a1cdd049-d5d5-4a3a-87cb-360491c56536-kube-api-access-s7d9k\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.511742 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.511713 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7d9k\" (UniqueName: \"kubernetes.io/projected/a1cdd049-d5d5-4a3a-87cb-360491c56536-kube-api-access-s7d9k\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.511869 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.511763 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1cdd049-d5d5-4a3a-87cb-360491c56536-metrics-certs\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.511869 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.511782 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1cdd049-d5d5-4a3a-87cb-360491c56536-maas-api-tls\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.511869 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.511812 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1cdd049-d5d5-4a3a-87cb-360491c56536-openshift-service-ca\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.512381 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.512361 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1cdd049-d5d5-4a3a-87cb-360491c56536-openshift-service-ca\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.514236 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.514213 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1cdd049-d5d5-4a3a-87cb-360491c56536-metrics-certs\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.514397 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.514373 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1cdd049-d5d5-4a3a-87cb-360491c56536-maas-api-tls\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.535528 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.535506 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7d9k\" (UniqueName: \"kubernetes.io/projected/a1cdd049-d5d5-4a3a-87cb-360491c56536-kube-api-access-s7d9k\") pod \"maas-api-bcf7db76f-srkgv\" (UID: \"a1cdd049-d5d5-4a3a-87cb-360491c56536\") " pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.629812 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.629754 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:42.928379 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:42.928295 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv"] Oct 29 22:48:42.933335 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:48:42.933309 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cdd049_d5d5_4a3a_87cb_360491c56536.slice/crio-a74ecf19297fb72e745a869e127f3e5445b87644d5559de335e5b0ec058b5bff WatchSource:0}: Error finding container a74ecf19297fb72e745a869e127f3e5445b87644d5559de335e5b0ec058b5bff: Status 404 returned error can't find the container with id a74ecf19297fb72e745a869e127f3e5445b87644d5559de335e5b0ec058b5bff Oct 29 22:48:43.370253 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:43.370212 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" event={"ID":"a1cdd049-d5d5-4a3a-87cb-360491c56536","Type":"ContainerStarted","Data":"a74ecf19297fb72e745a869e127f3e5445b87644d5559de335e5b0ec058b5bff"} Oct 29 22:48:44.376162 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:44.376125 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" event={"ID":"a1cdd049-d5d5-4a3a-87cb-360491c56536","Type":"ContainerStarted","Data":"3c6b7f5035585c06652f278698c5acfe020f329d1241b462832bd67222e4eedf"} Oct 29 22:48:44.376558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:44.376225 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:44.418001 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:44.417837 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" podStartSLOduration=1.976083809 podStartE2EDuration="2.417822672s" totalImagesPullingTime="441.738863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:44.417244021 +0000 UTC m=+321.533163630" watchObservedRunningTime="2026-10-29 22:48:44.417822672 +0000 UTC m=+321.533742268" Oct 29 22:48:50.385356 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.385323 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bcf7db76f-srkgv" Oct 29 22:48:50.456223 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.456188 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f695549b-54g6g"] Oct 29 22:48:50.456727 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.456692 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" podUID="c2597b61-cae0-4fb3-a81e-c482e4af83f7" containerName="maas-api" containerID="cri-o://24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c" gracePeriod=30 Oct 29 22:48:50.692225 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.692201 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:50.780560 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.780528 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-maas-api-tls\") pod \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " Oct 29 22:48:50.780759 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.780634 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2597b61-cae0-4fb3-a81e-c482e4af83f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2597b61-cae0-4fb3-a81e-c482e4af83f7-openshift-service-ca\") pod \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " Oct 29 22:48:50.780759 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.780668 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-metrics-certs\") pod \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " Oct 29 22:48:50.780759 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.780710 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2597b61-cae0-4fb3-a81e-c482e4af83f7-kube-api-access-m47nq\" (UniqueName: \"kubernetes.io/projected/c2597b61-cae0-4fb3-a81e-c482e4af83f7-kube-api-access-m47nq\") pod \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\" (UID: \"c2597b61-cae0-4fb3-a81e-c482e4af83f7\") " Oct 29 22:48:50.781005 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.780979 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2597b61-cae0-4fb3-a81e-c482e4af83f7-openshift-service-ca" pod "c2597b61-cae0-4fb3-a81e-c482e4af83f7" (UID: "c2597b61-cae0-4fb3-a81e-c482e4af83f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:50.782853 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.782829 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-maas-api-tls" pod "c2597b61-cae0-4fb3-a81e-c482e4af83f7" (UID: "c2597b61-cae0-4fb3-a81e-c482e4af83f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:50.782986 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.782961 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2597b61-cae0-4fb3-a81e-c482e4af83f7-kube-api-access-m47nq" pod "c2597b61-cae0-4fb3-a81e-c482e4af83f7" (UID: "c2597b61-cae0-4fb3-a81e-c482e4af83f7"). InnerVolumeSpecName "kube-api-access-m47nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:50.783024 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.782965 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-metrics-certs" pod "c2597b61-cae0-4fb3-a81e-c482e4af83f7" (UID: "c2597b61-cae0-4fb3-a81e-c482e4af83f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:50.881560 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.881527 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m47nq\" (UniqueName: \"kubernetes.io/projected/c2597b61-cae0-4fb3-a81e-c482e4af83f7-kube-api-access-m47nq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:50.881560 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.881556 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:50.881777 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.881570 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2597b61-cae0-4fb3-a81e-c482e4af83f7-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:50.881777 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:50.881604 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2597b61-cae0-4fb3-a81e-c482e4af83f7-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:48:51.405903 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:51.405817 5736 generic.go:356] "Generic (PLEG): container finished" podID="c2597b61-cae0-4fb3-a81e-c482e4af83f7" containerID="24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c" exitCode=0 Oct 29 22:48:51.405903 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:51.405870 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" Oct 29 22:48:51.406362 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:51.405902 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" event={"ID":"c2597b61-cae0-4fb3-a81e-c482e4af83f7","Type":"ContainerDied","Data":"24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c"} Oct 29 22:48:51.406362 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:51.405953 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f695549b-54g6g" event={"ID":"c2597b61-cae0-4fb3-a81e-c482e4af83f7","Type":"ContainerDied","Data":"4d996bcc52ec109fbf9b272bc79c379dc65510b25241c8dbd65519cd0b6f3f23"} Oct 29 22:48:51.406362 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:51.405975 5736 scope.go:122] "RemoveContainer" containerID="24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c" Oct 29 22:48:51.414459 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:51.414438 5736 scope.go:122] "RemoveContainer" containerID="24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c" Oct 29 22:48:51.414740 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:48:51.414723 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c\": container with ID starting with 24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c not found: ID does not exist" containerID="24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c" Oct 29 22:48:51.414799 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:51.414750 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c"} err="failed to get container status \"24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c\": rpc error: code = NotFound desc = could not find container \"24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c\": container with ID starting with 24139f87226c32866ada7bd242a0a6e543da3422e43b7c35826bddab212f237c not found: ID does not exist" Oct 29 22:48:51.451323 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:51.451298 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f695549b-54g6g"] Oct 29 22:48:51.461900 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:51.461876 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f695549b-54g6g"] Oct 29 22:48:53.001845 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:48:53.001805 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2597b61-cae0-4fb3-a81e-c482e4af83f7" path="/var/lib/kubelet/pods/c2597b61-cae0-4fb3-a81e-c482e4af83f7/volumes" Oct 29 22:48:56.822813 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:48:56.822769 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:49:05.598381 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.598344 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh"] Oct 29 22:49:05.598969 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.598916 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c2597b61-cae0-4fb3-a81e-c482e4af83f7" containerName="maas-api" Oct 29 22:49:05.598969 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.598941 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2597b61-cae0-4fb3-a81e-c482e4af83f7" containerName="maas-api" Oct 29 22:49:05.599167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.599016 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2597b61-cae0-4fb3-a81e-c482e4af83f7" containerName="maas-api" Oct 29 22:49:05.602071 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.602050 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:05.623630 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.623607 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh"] Oct 29 22:49:05.633308 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.633287 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 22:49:05.633308 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.633302 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 29 22:49:05.633437 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.633296 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 22:49:05.633437 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.633338 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jv9jd\"" Oct 29 22:49:05.704200 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.704170 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746ps\" (UniqueName: \"kubernetes.io/projected/5e30a728-ba3a-4405-8a8f-5b1ec88db691-kube-api-access-746ps\") pod \"llmisvc-controller-manager-6466c554d8-l9tmh\" (UID: \"5e30a728-ba3a-4405-8a8f-5b1ec88db691\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:05.704311 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.704211 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e30a728-ba3a-4405-8a8f-5b1ec88db691-cert\") pod \"llmisvc-controller-manager-6466c554d8-l9tmh\" (UID: \"5e30a728-ba3a-4405-8a8f-5b1ec88db691\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:05.804834 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.804809 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746ps\" (UniqueName: \"kubernetes.io/projected/5e30a728-ba3a-4405-8a8f-5b1ec88db691-kube-api-access-746ps\") pod \"llmisvc-controller-manager-6466c554d8-l9tmh\" (UID: \"5e30a728-ba3a-4405-8a8f-5b1ec88db691\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:05.804953 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.804849 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e30a728-ba3a-4405-8a8f-5b1ec88db691-cert\") pod \"llmisvc-controller-manager-6466c554d8-l9tmh\" (UID: \"5e30a728-ba3a-4405-8a8f-5b1ec88db691\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:05.807355 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.807332 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e30a728-ba3a-4405-8a8f-5b1ec88db691-cert\") pod \"llmisvc-controller-manager-6466c554d8-l9tmh\" (UID: \"5e30a728-ba3a-4405-8a8f-5b1ec88db691\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:05.829108 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.829083 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-746ps\" (UniqueName: \"kubernetes.io/projected/5e30a728-ba3a-4405-8a8f-5b1ec88db691-kube-api-access-746ps\") pod \"llmisvc-controller-manager-6466c554d8-l9tmh\" (UID: \"5e30a728-ba3a-4405-8a8f-5b1ec88db691\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:05.914153 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:05.914080 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:06.210620 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:06.210528 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh"] Oct 29 22:49:06.214902 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:49:06.214875 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e30a728_ba3a_4405_8a8f_5b1ec88db691.slice/crio-5c631be4d47dd9fb00ae072b107e71921ea5a935bf9a972e629b38f49f211f74 WatchSource:0}: Error finding container 5c631be4d47dd9fb00ae072b107e71921ea5a935bf9a972e629b38f49f211f74: Status 404 returned error can't find the container with id 5c631be4d47dd9fb00ae072b107e71921ea5a935bf9a972e629b38f49f211f74 Oct 29 22:49:06.471422 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:06.471346 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" event={"ID":"5e30a728-ba3a-4405-8a8f-5b1ec88db691","Type":"ContainerStarted","Data":"5c631be4d47dd9fb00ae072b107e71921ea5a935bf9a972e629b38f49f211f74"} Oct 29 22:49:08.481261 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:08.481227 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" event={"ID":"5e30a728-ba3a-4405-8a8f-5b1ec88db691","Type":"ContainerStarted","Data":"9a1123a82991ff98a1670b325a8d4ddda55a030658a88d94b3f34da814c41c36"} Oct 29 22:49:08.481652 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:08.481437 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:08.519211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:08.519163 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" podStartSLOduration=1.723646795 podStartE2EDuration="3.519152184s" totalImagesPullingTime="1.795505389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:08.518932778 +0000 UTC m=+345.634852384" watchObservedRunningTime="2026-10-29 22:49:08.519152184 +0000 UTC m=+345.635071787" Oct 29 22:49:26.823321 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:49:26.823285 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:49:39.487069 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:49:39.486990 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l9tmh" Oct 29 22:49:56.821974 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:49:56.821938 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:50:26.822078 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:50:26.822038 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:50:49.668011 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.667967 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g"] Oct 29 22:50:49.671929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.671905 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.696028 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.695745 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-28mq9\"" Oct 29 22:50:49.696028 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.695816 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 29 22:50:49.696028 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.695853 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 29 22:50:49.696028 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.695907 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 29 22:50:49.697153 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.697130 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g"] Oct 29 22:50:49.741380 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.741349 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.741504 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.741392 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff018793-fe72-4680-af0f-6a7b1d4b94b1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.741504 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.741423 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.741504 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.741466 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlvp\" (UniqueName: \"kubernetes.io/projected/ff018793-fe72-4680-af0f-6a7b1d4b94b1-kube-api-access-pdlvp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.741653 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.741630 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.741727 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.741714 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.842780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.842743 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.842929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.842790 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff018793-fe72-4680-af0f-6a7b1d4b94b1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.842929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.842810 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.842929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.842828 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlvp\" (UniqueName: \"kubernetes.io/projected/ff018793-fe72-4680-af0f-6a7b1d4b94b1-kube-api-access-pdlvp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.842929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.842862 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.842929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.842890 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.843215 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.843195 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.843259 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.843213 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.843293 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.843262 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.845197 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.845169 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff018793-fe72-4680-af0f-6a7b1d4b94b1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.845383 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.845364 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff018793-fe72-4680-af0f-6a7b1d4b94b1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:49.864277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:49.864247 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlvp\" (UniqueName: \"kubernetes.io/projected/ff018793-fe72-4680-af0f-6a7b1d4b94b1-kube-api-access-pdlvp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g\" (UID: \"ff018793-fe72-4680-af0f-6a7b1d4b94b1\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:50.004132 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:50.004100 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:50.306496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:50.306425 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g"] Oct 29 22:50:50.311423 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:50:50.311398 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff018793_fe72_4680_af0f_6a7b1d4b94b1.slice/crio-e9bab30a1ea65c54345b5053dfacae1784229e7c9527ed9c4a8cbf02b269a7ed WatchSource:0}: Error finding container e9bab30a1ea65c54345b5053dfacae1784229e7c9527ed9c4a8cbf02b269a7ed: Status 404 returned error can't find the container with id e9bab30a1ea65c54345b5053dfacae1784229e7c9527ed9c4a8cbf02b269a7ed Oct 29 22:50:50.312862 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:50.312843 5736 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:50:50.915093 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:50.915055 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" event={"ID":"ff018793-fe72-4680-af0f-6a7b1d4b94b1","Type":"ContainerStarted","Data":"e9bab30a1ea65c54345b5053dfacae1784229e7c9527ed9c4a8cbf02b269a7ed"} Oct 29 22:50:51.921922 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:51.921886 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" event={"ID":"ff018793-fe72-4680-af0f-6a7b1d4b94b1","Type":"ContainerStarted","Data":"99e7cbb8f51595ee09b36421e19af0491a80727b0b81b2c95c558af3dacfb06f"} Oct 29 22:50:51.960945 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:51.960902 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podStartSLOduration=1.885428808 podStartE2EDuration="2.960889555s" totalImagesPullingTime="1.075460747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:51.960190209 +0000 UTC m=+449.076109814" watchObservedRunningTime="2026-10-29 22:50:51.960889555 +0000 UTC m=+449.076809161" Oct 29 22:50:52.004389 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:50:52.004365 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:50:56.823412 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:50:56.823373 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:51:00.004909 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:00.004879 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:51:00.018112 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:00.018090 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:51:00.971692 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:00.971659 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:51:03.193109 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.193075 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz"] Oct 29 22:51:03.196945 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.196913 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.210571 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.210549 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 22:51:03.224422 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.224393 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz"] Oct 29 22:51:03.301760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.301732 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h76hm\"" Oct 29 22:51:03.366112 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.366072 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.366268 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.366118 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.366268 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.366211 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.366268 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.366242 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sk6b\" (UniqueName: \"kubernetes.io/projected/98a25bcb-0734-4d38-a889-214cfcf54e8b-kube-api-access-2sk6b\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.366435 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.366349 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.366435 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.366390 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.366521 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.366486 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a25bcb-0734-4d38-a889-214cfcf54e8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.366521 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.366515 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.366620 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.366548 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.466951 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.466918 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a25bcb-0734-4d38-a889-214cfcf54e8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467137 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.466959 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467137 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467073 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467265 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467174 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467265 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467207 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467265 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467245 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467415 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467271 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sk6b\" (UniqueName: \"kubernetes.io/projected/98a25bcb-0734-4d38-a889-214cfcf54e8b-kube-api-access-2sk6b\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467415 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467309 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467415 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467316 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467415 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467333 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467415 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467381 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467745 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467728 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467806 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467781 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a25bcb-0734-4d38-a889-214cfcf54e8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.467848 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.467815 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.469679 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.469652 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.469969 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.469949 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.489010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.488983 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a25bcb-0734-4d38-a889-214cfcf54e8b-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.489454 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.489433 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sk6b\" (UniqueName: \"kubernetes.io/projected/98a25bcb-0734-4d38-a889-214cfcf54e8b-kube-api-access-2sk6b\") pod \"maas-default-gateway-openshift-default-55fbd65575-2dlnz\" (UID: \"98a25bcb-0734-4d38-a889-214cfcf54e8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.509974 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.509953 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:03.829836 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.829716 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz"] Oct 29 22:51:03.834194 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:51:03.834151 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a25bcb_0734_4d38_a889_214cfcf54e8b.slice/crio-c66bddcafddb792a83e0ef07137589d9ac746322d50e865709c58ffb2000f7d2 WatchSource:0}: Error finding container c66bddcafddb792a83e0ef07137589d9ac746322d50e865709c58ffb2000f7d2: Status 404 returned error can't find the container with id c66bddcafddb792a83e0ef07137589d9ac746322d50e865709c58ffb2000f7d2 Oct 29 22:51:03.975681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:03.975638 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" event={"ID":"98a25bcb-0734-4d38-a889-214cfcf54e8b","Type":"ContainerStarted","Data":"c66bddcafddb792a83e0ef07137589d9ac746322d50e865709c58ffb2000f7d2"} Oct 29 22:51:06.422493 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:06.422460 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:06.422763 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:06.422531 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:06.422763 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:06.422561 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:06.990908 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:06.990871 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" event={"ID":"98a25bcb-0734-4d38-a889-214cfcf54e8b","Type":"ContainerStarted","Data":"0230c6bfda75eca0b089d31a8ff921aaffce56586509c2562b0a3e44f4ca2aaf"} Oct 29 22:51:07.055870 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:07.054634 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" podStartSLOduration=1.469297247 podStartE2EDuration="4.054611766s" totalImagesPullingTime="2.585314519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:07.050245192 +0000 UTC m=+464.166164800" watchObservedRunningTime="2026-10-29 22:51:07.054611766 +0000 UTC m=+464.170531376" Oct 29 22:51:07.511062 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:07.511023 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:07.512531 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:07.512502 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2dlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 29 22:51:07.512675 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:07.512553 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" podUID="98a25bcb-0734-4d38-a889-214cfcf54e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 29 22:51:08.511106 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:08.511073 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2dlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 29 22:51:08.511564 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:08.511148 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" podUID="98a25bcb-0734-4d38-a889-214cfcf54e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 29 22:51:09.510323 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:09.510292 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2dlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 29 22:51:09.510498 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:09.510339 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" podUID="98a25bcb-0734-4d38-a889-214cfcf54e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 29 22:51:10.510946 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:10.510912 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2dlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 29 22:51:10.511390 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:10.510963 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" podUID="98a25bcb-0734-4d38-a889-214cfcf54e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 29 22:51:11.510734 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:11.510689 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2dlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 29 22:51:11.510925 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:11.510768 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" podUID="98a25bcb-0734-4d38-a889-214cfcf54e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 29 22:51:12.510225 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:12.510192 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2dlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 29 22:51:12.510650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:12.510267 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" podUID="98a25bcb-0734-4d38-a889-214cfcf54e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 29 22:51:13.510740 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:13.510705 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2dlnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Oct 29 22:51:13.511144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:13.510758 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" podUID="98a25bcb-0734-4d38-a889-214cfcf54e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Oct 29 22:51:14.575313 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:14.575279 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:14.575816 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:14.575651 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:14.576350 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:14.576327 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" Oct 29 22:51:18.209344 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.209310 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v"] Oct 29 22:51:18.212969 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.212947 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.219201 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.219177 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.219320 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.219211 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.219400 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.219362 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.219956 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.219930 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg2z5\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-kube-api-access-sg2z5\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.220046 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.219974 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.220149 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.220121 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.220201 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.220163 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d23e612-818a-43b7-9ff1-c3d37820460f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.220270 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.220248 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.220312 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.220283 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.238920 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.238896 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v"] Oct 29 22:51:18.320741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.320712 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d23e612-818a-43b7-9ff1-c3d37820460f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.320879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.320749 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.320879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.320765 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.320879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.320806 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.320879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.320820 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.321090 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.320962 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.321090 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.321011 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2z5\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-kube-api-access-sg2z5\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.321090 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.321041 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.321090 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.321080 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.321318 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.321292 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.321379 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.321313 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.321417 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.321396 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.321451 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.321407 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.321597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.321555 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d23e612-818a-43b7-9ff1-c3d37820460f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.323327 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.323303 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.323422 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.323393 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.342554 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.342526 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.342874 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.342853 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg2z5\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-kube-api-access-sg2z5\") pod \"maas-default-gateway-openshift-default-55fbd65575-ff84v\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.526014 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.525937 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:18.842023 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.841609 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v"] Oct 29 22:51:18.845849 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:51:18.845826 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d23e612_818a_43b7_9ff1_c3d37820460f.slice/crio-aa1726ff71815ad85950c7062c59436c0dd35434ead63c248d6f28fb1443f64c WatchSource:0}: Error finding container aa1726ff71815ad85950c7062c59436c0dd35434ead63c248d6f28fb1443f64c: Status 404 returned error can't find the container with id aa1726ff71815ad85950c7062c59436c0dd35434ead63c248d6f28fb1443f64c Oct 29 22:51:18.848807 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.848771 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:18.848883 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.848841 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:18.848883 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:18.848873 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:19.044035 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:19.043994 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" event={"ID":"2d23e612-818a-43b7-9ff1-c3d37820460f","Type":"ContainerStarted","Data":"2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6"} Oct 29 22:51:19.044222 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:19.044061 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" event={"ID":"2d23e612-818a-43b7-9ff1-c3d37820460f","Type":"ContainerStarted","Data":"aa1726ff71815ad85950c7062c59436c0dd35434ead63c248d6f28fb1443f64c"} Oct 29 22:51:19.087148 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:19.087075 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podStartSLOduration=1.087057575 podStartE2EDuration="1.087057575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:19.086151304 +0000 UTC m=+476.202070910" watchObservedRunningTime="2026-10-29 22:51:19.087057575 +0000 UTC m=+476.202977194" Oct 29 22:51:19.526993 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:19.526958 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:19.528329 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:19.528304 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 29 22:51:19.528429 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:19.528351 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 29 22:51:20.526558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:20.526527 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 29 22:51:20.526746 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:20.526602 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 29 22:51:21.526930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:21.526894 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 29 22:51:21.527414 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:21.526952 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 29 22:51:22.526497 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:22.526465 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 29 22:51:22.526694 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:22.526522 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 29 22:51:23.526936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:23.526899 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 29 22:51:23.527320 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:23.526958 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 29 22:51:24.526258 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:24.526223 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 29 22:51:24.526428 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:24.526277 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 29 22:51:25.527097 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:25.527059 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 29 22:51:25.527484 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:25.527135 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 29 22:51:26.822649 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:51:26.822610 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:51:27.156798 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:27.156705 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:27.157177 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:27.157150 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:27.158094 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:27.158062 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 22:51:33.179338 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.179309 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6"] Oct 29 22:51:33.184796 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.184775 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.206635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.206610 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6"] Oct 29 22:51:33.270603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.270546 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.270761 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.270623 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.270761 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.270646 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.270761 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.270684 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.270889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.270811 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596e957b-f674-4a0e-92eb-08913a45b069-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.270889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.270875 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.270966 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.270895 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.271020 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.271001 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hc8q\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-kube-api-access-6hc8q\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.271057 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.271046 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596e957b-f674-4a0e-92eb-08913a45b069-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.371551 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.371522 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.371551 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.371554 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.371792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.371598 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hc8q\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-kube-api-access-6hc8q\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.371792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.371633 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596e957b-f674-4a0e-92eb-08913a45b069-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.371792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.371661 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.371792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.371691 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.371792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.371714 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.371792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.371756 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.371792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.371787 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596e957b-f674-4a0e-92eb-08913a45b069-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.372111 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.372075 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.372239 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.372211 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.372359 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.372304 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.372433 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.372375 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.372496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.372442 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596e957b-f674-4a0e-92eb-08913a45b069-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.373876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.373854 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.374679 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.374662 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596e957b-f674-4a0e-92eb-08913a45b069-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.393303 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.393280 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.393599 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.393560 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hc8q\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-kube-api-access-6hc8q\") pod \"maas-default-gateway-openshift-default-55fbd65575-qmkl6\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.496983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.496960 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:33.813761 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.813683 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6"] Oct 29 22:51:33.817941 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:51:33.817913 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596e957b_f674_4a0e_92eb_08913a45b069.slice/crio-eb143e6c3789d3a0a932187b53156e701a90d829956409ac156fc40d77a684b4 WatchSource:0}: Error finding container eb143e6c3789d3a0a932187b53156e701a90d829956409ac156fc40d77a684b4: Status 404 returned error can't find the container with id eb143e6c3789d3a0a932187b53156e701a90d829956409ac156fc40d77a684b4 Oct 29 22:51:33.820012 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.819980 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:33.820071 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.820043 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:33.820104 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:33.820073 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:34.123572 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:34.123496 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" event={"ID":"596e957b-f674-4a0e-92eb-08913a45b069","Type":"ContainerStarted","Data":"6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f"} Oct 29 22:51:34.123572 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:34.123545 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" event={"ID":"596e957b-f674-4a0e-92eb-08913a45b069","Type":"ContainerStarted","Data":"eb143e6c3789d3a0a932187b53156e701a90d829956409ac156fc40d77a684b4"} Oct 29 22:51:34.165727 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:34.165675 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podStartSLOduration=1.165661712 podStartE2EDuration="1.165661712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:34.16490482 +0000 UTC m=+491.280824427" watchObservedRunningTime="2026-10-29 22:51:34.165661712 +0000 UTC m=+491.281581317" Oct 29 22:51:34.497857 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:34.497828 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:34.499209 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:34.499184 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 29 22:51:34.499330 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:34.499245 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 29 22:51:35.497741 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:35.497696 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 29 22:51:35.497905 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:35.497761 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 29 22:51:36.497887 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:36.497853 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 29 22:51:36.498068 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:36.497909 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 29 22:51:37.281680 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.281643 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt"] Oct 29 22:51:37.287974 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.287938 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.313498 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.313467 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e57216-openshift-default-dockercfg-jh4hn\"" Oct 29 22:51:37.323380 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.323356 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt"] Oct 29 22:51:37.408996 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.408960 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.409154 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.409002 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.409154 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.409108 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.409230 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.409177 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78a262-e18e-4ec5-ae50-226bc71789a1-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.409265 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.409225 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.409297 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.409276 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.409329 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.409297 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnnfv\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-kube-api-access-tnnfv\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.409409 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.409396 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.409442 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.409422 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.497838 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.497805 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 29 22:51:37.497999 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.497857 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 29 22:51:37.510260 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510223 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.510681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510274 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnnfv\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-kube-api-access-tnnfv\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.510681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510362 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.510681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510405 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.510681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510456 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.510681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510492 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.510681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510527 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.510681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510565 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78a262-e18e-4ec5-ae50-226bc71789a1-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.510681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510628 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.511079 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510778 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.511079 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510886 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.511079 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.510963 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.511079 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.511059 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.511422 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.511403 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78a262-e18e-4ec5-ae50-226bc71789a1-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.512492 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.512470 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.513311 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.513295 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.541203 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.541128 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.564280 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.564242 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnnfv\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-kube-api-access-tnnfv\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.605155 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.605120 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:37.961651 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.960914 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt"] Oct 29 22:51:37.965281 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:51:37.965255 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef78a262_e18e_4ec5_ae50_226bc71789a1.slice/crio-e50aec1ba6122d07a459d9c9ee6c78ad9558548524caefb100afb9656c1b812a WatchSource:0}: Error finding container e50aec1ba6122d07a459d9c9ee6c78ad9558548524caefb100afb9656c1b812a: Status 404 returned error can't find the container with id e50aec1ba6122d07a459d9c9ee6c78ad9558548524caefb100afb9656c1b812a Oct 29 22:51:37.967689 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.967651 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:37.967790 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.967736 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:37.967871 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:37.967810 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:38.174323 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:38.174286 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" event={"ID":"ef78a262-e18e-4ec5-ae50-226bc71789a1","Type":"ContainerStarted","Data":"fadbc2901818a85a4df9a69d5944cec3b8612a17b5678e5927ba2dd467fcaae4"} Oct 29 22:51:38.174474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:38.174353 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" event={"ID":"ef78a262-e18e-4ec5-ae50-226bc71789a1","Type":"ContainerStarted","Data":"e50aec1ba6122d07a459d9c9ee6c78ad9558548524caefb100afb9656c1b812a"} Oct 29 22:51:38.222903 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:38.222822 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" podStartSLOduration=1.222809629 podStartE2EDuration="1.222809629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:38.222192831 +0000 UTC m=+495.338112438" watchObservedRunningTime="2026-10-29 22:51:38.222809629 +0000 UTC m=+495.338729236" Oct 29 22:51:38.498396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:38.498316 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 29 22:51:38.498396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:38.498370 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 29 22:51:38.605270 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:38.605233 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:39.353573 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.353535 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2"] Oct 29 22:51:39.358840 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.358814 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.389523 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.389483 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5db969e0-serving-cert\"" Oct 29 22:51:39.390923 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.390899 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2"] Oct 29 22:51:39.429016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.428988 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-metrics-certs\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.429147 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.429030 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ca785-8c0a-4288-a1e0-346e7e770822-openshift-service-ca\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.429222 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.429201 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.429273 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.429241 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvx24\" (UniqueName: \"kubernetes.io/projected/254ca785-8c0a-4288-a1e0-346e7e770822-kube-api-access-rvx24\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.497645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.497617 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 29 22:51:39.497795 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.497665 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 29 22:51:39.530454 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.530429 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.530614 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.530465 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvx24\" (UniqueName: \"kubernetes.io/projected/254ca785-8c0a-4288-a1e0-346e7e770822-kube-api-access-rvx24\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.530614 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:51:39.530574 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-serving-cert: secret "maas-api-e2e-ait-5db969e0-serving-cert" not found Oct 29 22:51:39.530735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.530605 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-metrics-certs\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.530735 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:51:39.530672 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls podName:254ca785-8c0a-4288-a1e0-346e7e770822 nodeName:}" failed. No retries permitted until 2026-10-29 22:51:40.03065125 +0000 UTC m=+497.146570849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls") pod "maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" (UID: "254ca785-8c0a-4288-a1e0-346e7e770822") : secret "maas-api-e2e-ait-5db969e0-serving-cert" not found Oct 29 22:51:39.530735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.530695 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ca785-8c0a-4288-a1e0-346e7e770822-openshift-service-ca\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.558605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.531500 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ca785-8c0a-4288-a1e0-346e7e770822-openshift-service-ca\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.562150 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.562109 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-metrics-certs\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.562408 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.562383 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvx24\" (UniqueName: \"kubernetes.io/projected/254ca785-8c0a-4288-a1e0-346e7e770822-kube-api-access-rvx24\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:39.609941 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:39.609882 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:40.035454 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:40.035424 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:40.037868 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:40.037849 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls\") pod \"maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:40.182856 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:40.182826 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:40.183874 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:40.183847 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:51:40.271886 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:40.271861 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:40.498519 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:40.498475 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 29 22:51:40.498727 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:40.498524 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 29 22:51:40.580131 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:40.580041 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2"] Oct 29 22:51:40.583992 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:51:40.583964 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254ca785_8c0a_4288_a1e0_346e7e770822.slice/crio-868d6d3026a0a34d565da909f382049f1294009bb28c62eefd5032899b69b19b WatchSource:0}: Error finding container 868d6d3026a0a34d565da909f382049f1294009bb28c62eefd5032899b69b19b: Status 404 returned error can't find the container with id 868d6d3026a0a34d565da909f382049f1294009bb28c62eefd5032899b69b19b Oct 29 22:51:41.195173 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.195076 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" event={"ID":"254ca785-8c0a-4288-a1e0-346e7e770822","Type":"ContainerStarted","Data":"82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf"} Oct 29 22:51:41.195173 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.195131 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" event={"ID":"254ca785-8c0a-4288-a1e0-346e7e770822","Type":"ContainerStarted","Data":"868d6d3026a0a34d565da909f382049f1294009bb28c62eefd5032899b69b19b"} Oct 29 22:51:41.237872 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.237814 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" podStartSLOduration=1.8784926610000001 podStartE2EDuration="2.237796608s" totalImagesPullingTime="359.303947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:41.237558496 +0000 UTC m=+498.353478102" watchObservedRunningTime="2026-10-29 22:51:41.237796608 +0000 UTC m=+498.353716216" Oct 29 22:51:41.462967 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.462853 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl"] Oct 29 22:51:41.466607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.466575 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.504436 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.503694 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl"] Oct 29 22:51:41.544895 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.544864 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-2bef9ca4-serving-cert\"" Oct 29 22:51:41.551061 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.551030 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5j4t\" (UniqueName: \"kubernetes.io/projected/5f9149be-f448-4a65-b69b-bef926de3233-kube-api-access-c5j4t\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.551243 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.551078 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-metrics-certs\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.551243 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.551157 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.551243 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.551223 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9149be-f448-4a65-b69b-bef926de3233-openshift-service-ca\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.652796 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.652746 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5j4t\" (UniqueName: \"kubernetes.io/projected/5f9149be-f448-4a65-b69b-bef926de3233-kube-api-access-c5j4t\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.652796 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.652810 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-metrics-certs\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.653082 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.652847 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.653082 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:51:41.652970 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-serving-cert: secret "maas-api-e2e-migrate-2bef9ca4-serving-cert" not found Oct 29 22:51:41.653082 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:51:41.653035 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls podName:5f9149be-f448-4a65-b69b-bef926de3233 nodeName:}" failed. No retries permitted until 2026-10-29 22:51:42.153019577 +0000 UTC m=+499.268939178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls") pod "maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" (UID: "5f9149be-f448-4a65-b69b-bef926de3233") : secret "maas-api-e2e-migrate-2bef9ca4-serving-cert" not found Oct 29 22:51:41.653082 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.653026 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9149be-f448-4a65-b69b-bef926de3233-openshift-service-ca\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.653747 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.653724 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9149be-f448-4a65-b69b-bef926de3233-openshift-service-ca\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.655427 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.655403 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-metrics-certs\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:41.675649 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:41.675615 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5j4t\" (UniqueName: \"kubernetes.io/projected/5f9149be-f448-4a65-b69b-bef926de3233-kube-api-access-c5j4t\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:42.133125 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:42.133081 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:42.136009 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:42.134079 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:42.136396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:42.136379 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 22:51:42.158855 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:42.158823 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:42.161421 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:42.161397 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls\") pod \"maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:42.200408 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:42.200373 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:42.389841 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:42.389759 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:42.703857 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:42.703777 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl"] Oct 29 22:51:42.708292 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:51:42.708263 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9149be_f448_4a65_b69b_bef926de3233.slice/crio-735032c104d09aa06cf22680aaee73ad27f7038bb96f8b5abd7b57cbd3efb1c3 WatchSource:0}: Error finding container 735032c104d09aa06cf22680aaee73ad27f7038bb96f8b5abd7b57cbd3efb1c3: Status 404 returned error can't find the container with id 735032c104d09aa06cf22680aaee73ad27f7038bb96f8b5abd7b57cbd3efb1c3 Oct 29 22:51:43.205963 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.205923 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" event={"ID":"5f9149be-f448-4a65-b69b-bef926de3233","Type":"ContainerStarted","Data":"ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a"} Oct 29 22:51:43.206422 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.205988 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" event={"ID":"5f9149be-f448-4a65-b69b-bef926de3233","Type":"ContainerStarted","Data":"735032c104d09aa06cf22680aaee73ad27f7038bb96f8b5abd7b57cbd3efb1c3"} Oct 29 22:51:43.206422 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.206108 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:43.247306 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.247251 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" podStartSLOduration=1.896450415 podStartE2EDuration="2.247234252s" totalImagesPullingTime="350.783837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:43.246293124 +0000 UTC m=+500.362212731" watchObservedRunningTime="2026-10-29 22:51:43.247234252 +0000 UTC m=+500.363153859" Oct 29 22:51:43.527421 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.527387 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6"] Oct 29 22:51:43.531083 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.531062 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.544502 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.544482 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e406071d-serving-cert\"" Oct 29 22:51:43.553257 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.553230 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6"] Oct 29 22:51:43.575645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.575616 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkct\" (UniqueName: \"kubernetes.io/projected/8f292623-0627-41f2-974a-239ff5b3e739-kube-api-access-mfkct\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.575791 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.575665 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.575855 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.575831 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f292623-0627-41f2-974a-239ff5b3e739-openshift-service-ca\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.575898 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.575878 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-metrics-certs\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.676981 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.676943 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkct\" (UniqueName: \"kubernetes.io/projected/8f292623-0627-41f2-974a-239ff5b3e739-kube-api-access-mfkct\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.676981 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.676985 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.677225 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.677012 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f292623-0627-41f2-974a-239ff5b3e739-openshift-service-ca\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.677225 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.677030 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-metrics-certs\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.677225 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:51:43.677127 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-serving-cert: secret "maas-api-e2e-derive-e406071d-serving-cert" not found Oct 29 22:51:43.677225 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:51:43.677211 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls podName:8f292623-0627-41f2-974a-239ff5b3e739 nodeName:}" failed. No retries permitted until 2026-10-29 22:51:44.177186779 +0000 UTC m=+501.293106378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls") pod "maas-api-e2e-derive-e406071d-779d879f97-wvxn6" (UID: "8f292623-0627-41f2-974a-239ff5b3e739") : secret "maas-api-e2e-derive-e406071d-serving-cert" not found Oct 29 22:51:43.677746 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.677724 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f292623-0627-41f2-974a-239ff5b3e739-openshift-service-ca\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.679324 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.679302 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-metrics-certs\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:43.700423 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:43.700394 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkct\" (UniqueName: \"kubernetes.io/projected/8f292623-0627-41f2-974a-239ff5b3e739-kube-api-access-mfkct\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:44.182548 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:44.182514 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:44.185008 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:44.184985 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls\") pod \"maas-api-e2e-derive-e406071d-779d879f97-wvxn6\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:44.444162 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:44.444070 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:44.769272 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:44.769229 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6"] Oct 29 22:51:44.773706 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:51:44.773668 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f292623_0627_41f2_974a_239ff5b3e739.slice/crio-2c9ff8936149847238f443522fcc9e8c959f485b59d402f553ae5fb124db88df WatchSource:0}: Error finding container 2c9ff8936149847238f443522fcc9e8c959f485b59d402f553ae5fb124db88df: Status 404 returned error can't find the container with id 2c9ff8936149847238f443522fcc9e8c959f485b59d402f553ae5fb124db88df Oct 29 22:51:45.216062 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:45.216032 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" event={"ID":"8f292623-0627-41f2-974a-239ff5b3e739","Type":"ContainerStarted","Data":"395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722"} Oct 29 22:51:45.216207 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:45.216082 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" event={"ID":"8f292623-0627-41f2-974a-239ff5b3e739","Type":"ContainerStarted","Data":"2c9ff8936149847238f443522fcc9e8c959f485b59d402f553ae5fb124db88df"} Oct 29 22:51:45.216207 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:45.216134 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:45.258136 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:45.258091 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" podStartSLOduration=1.910600386 podStartE2EDuration="2.258080363s" totalImagesPullingTime="347.479977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:45.257327669 +0000 UTC m=+502.373247275" watchObservedRunningTime="2026-10-29 22:51:45.258080363 +0000 UTC m=+502.373999968" Oct 29 22:51:48.022983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.022948 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8"] Oct 29 22:51:48.026756 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.026733 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.039789 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.039766 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-865d946f-serving-cert\"" Oct 29 22:51:48.051706 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.051679 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8"] Oct 29 22:51:48.120024 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.119991 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-metrics-certs\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.120179 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.120080 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-maas-api-tls\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.120217 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.120182 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwjq\" (UniqueName: \"kubernetes.io/projected/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-kube-api-access-sfwjq\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.120254 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.120231 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-openshift-service-ca\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.211692 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.211668 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:51:48.221106 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.221080 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-metrics-certs\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.221231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.221123 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-maas-api-tls\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.221231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.221192 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwjq\" (UniqueName: \"kubernetes.io/projected/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-kube-api-access-sfwjq\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.221231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.221213 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-openshift-service-ca\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.221860 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.221830 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-openshift-service-ca\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.223597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.223558 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-maas-api-tls\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.223597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.223569 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-metrics-certs\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.248938 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.248910 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwjq\" (UniqueName: \"kubernetes.io/projected/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-kube-api-access-sfwjq\") pod \"maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.339614 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.339518 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:48.676034 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:48.675954 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8"] Oct 29 22:51:48.681140 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:51:48.681104 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f51d2ba_5e4f_40fb_bc3c_f21edbcfcc3c.slice/crio-039ada2dc613a9e17decd2b4afc9d124a8fca2cd3b30b0d6d7d7dcf2c576ec9b WatchSource:0}: Error finding container 039ada2dc613a9e17decd2b4afc9d124a8fca2cd3b30b0d6d7d7dcf2c576ec9b: Status 404 returned error can't find the container with id 039ada2dc613a9e17decd2b4afc9d124a8fca2cd3b30b0d6d7d7dcf2c576ec9b Oct 29 22:51:49.216534 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:49.216506 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:51:49.238113 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:49.238077 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" event={"ID":"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c","Type":"ContainerStarted","Data":"778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb"} Oct 29 22:51:49.238292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:49.238142 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" event={"ID":"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c","Type":"ContainerStarted","Data":"039ada2dc613a9e17decd2b4afc9d124a8fca2cd3b30b0d6d7d7dcf2c576ec9b"} Oct 29 22:51:49.238292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:49.238192 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:49.297078 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:49.296945 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" podStartSLOduration=0.987073724 podStartE2EDuration="1.296924795s" totalImagesPullingTime="309.851071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:49.295897177 +0000 UTC m=+506.411816783" watchObservedRunningTime="2026-10-29 22:51:49.296924795 +0000 UTC m=+506.412844401" Oct 29 22:51:51.226167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:51.226138 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:51:52.473302 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.473263 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn"] Oct 29 22:51:52.477372 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.477350 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.492387 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.492369 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8042816e-serving-cert\"" Oct 29 22:51:52.504217 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.504187 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn"] Oct 29 22:51:52.566469 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.566440 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-maas-api-tls\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.566641 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.566551 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkmcw\" (UniqueName: \"kubernetes.io/projected/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-kube-api-access-hkmcw\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.566699 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.566637 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-metrics-certs\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.566699 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.566695 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-openshift-service-ca\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.667889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.667852 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-openshift-service-ca\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.668043 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.667894 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-maas-api-tls\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.668043 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.667985 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkmcw\" (UniqueName: \"kubernetes.io/projected/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-kube-api-access-hkmcw\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.668043 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.668039 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-metrics-certs\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.668661 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.668635 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-openshift-service-ca\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.671480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.671451 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-metrics-certs\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.673939 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.673919 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-maas-api-tls\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.692400 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.692378 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkmcw\" (UniqueName: \"kubernetes.io/projected/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-kube-api-access-hkmcw\") pod \"maas-api-e2e-ait-8042816e-6f86f79447-lcxwn\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:52.790432 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:52.790365 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:53.094382 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:53.094293 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn"] Oct 29 22:51:53.098516 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:51:53.098487 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb19ec2_d9ee_4c3c_9525_4f758c09a5e0.slice/crio-da1cfdb815e70fa9673da60a7b916f64a98b26a3633d6a4fbba74cf38935a6f6 WatchSource:0}: Error finding container da1cfdb815e70fa9673da60a7b916f64a98b26a3633d6a4fbba74cf38935a6f6: Status 404 returned error can't find the container with id da1cfdb815e70fa9673da60a7b916f64a98b26a3633d6a4fbba74cf38935a6f6 Oct 29 22:51:53.261156 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:53.261121 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" event={"ID":"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0","Type":"ContainerStarted","Data":"da1cfdb815e70fa9673da60a7b916f64a98b26a3633d6a4fbba74cf38935a6f6"} Oct 29 22:51:54.267557 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:54.267522 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" event={"ID":"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0","Type":"ContainerStarted","Data":"78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85"} Oct 29 22:51:54.268112 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:54.267607 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:51:54.306843 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:54.306792 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" podStartSLOduration=2.013437697 podStartE2EDuration="2.306775027s" totalImagesPullingTime="293.33733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:54.305389027 +0000 UTC m=+511.421308633" watchObservedRunningTime="2026-10-29 22:51:54.306775027 +0000 UTC m=+511.422694634" Oct 29 22:51:55.250118 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:51:55.250083 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:51:56.823007 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:51:56.822964 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:52:00.278171 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:00.278144 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:52:11.224730 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:11.224681 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:52:11.225190 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:11.224759 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:52:12.383155 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:12.383120 5736 generic.go:356] "Generic (PLEG): container finished" podID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerID="fadbc2901818a85a4df9a69d5944cec3b8612a17b5678e5927ba2dd467fcaae4" exitCode=0 Oct 29 22:52:12.383558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:12.383183 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" event={"ID":"ef78a262-e18e-4ec5-ae50-226bc71789a1","Type":"ContainerDied","Data":"fadbc2901818a85a4df9a69d5944cec3b8612a17b5678e5927ba2dd467fcaae4"} Oct 29 22:52:12.383558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:12.383503 5736 scope.go:122] "RemoveContainer" containerID="fadbc2901818a85a4df9a69d5944cec3b8612a17b5678e5927ba2dd467fcaae4" Oct 29 22:52:12.384686 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:12.384653 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:52:12.384736 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:12.384726 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:52:12.384771 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:12.384760 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:52:12.740744 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:12.740712 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl"] Oct 29 22:52:12.740983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:12.740953 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" podUID="5f9149be-f448-4a65-b69b-bef926de3233" containerName="maas-api" containerID="cri-o://ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a" gracePeriod=30 Oct 29 22:52:13.029823 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.029796 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:52:13.071760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.071715 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9149be-f448-4a65-b69b-bef926de3233-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9149be-f448-4a65-b69b-bef926de3233-openshift-service-ca\") pod \"5f9149be-f448-4a65-b69b-bef926de3233\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " Oct 29 22:52:13.071760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.071767 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls\") pod \"5f9149be-f448-4a65-b69b-bef926de3233\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " Oct 29 22:52:13.071978 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.071800 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-metrics-certs\") pod \"5f9149be-f448-4a65-b69b-bef926de3233\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " Oct 29 22:52:13.071978 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.071936 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9149be-f448-4a65-b69b-bef926de3233-kube-api-access-c5j4t\" (UniqueName: \"kubernetes.io/projected/5f9149be-f448-4a65-b69b-bef926de3233-kube-api-access-c5j4t\") pod \"5f9149be-f448-4a65-b69b-bef926de3233\" (UID: \"5f9149be-f448-4a65-b69b-bef926de3233\") " Oct 29 22:52:13.072204 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.072164 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9149be-f448-4a65-b69b-bef926de3233-openshift-service-ca" pod "5f9149be-f448-4a65-b69b-bef926de3233" (UID: "5f9149be-f448-4a65-b69b-bef926de3233"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:13.072360 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.072320 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9149be-f448-4a65-b69b-bef926de3233-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:13.073987 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.073963 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls" pod "5f9149be-f448-4a65-b69b-bef926de3233" (UID: "5f9149be-f448-4a65-b69b-bef926de3233"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:13.074881 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.074863 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9149be-f448-4a65-b69b-bef926de3233-kube-api-access-c5j4t" pod "5f9149be-f448-4a65-b69b-bef926de3233" (UID: "5f9149be-f448-4a65-b69b-bef926de3233"). InnerVolumeSpecName "kube-api-access-c5j4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:13.075092 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.075076 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-metrics-certs" pod "5f9149be-f448-4a65-b69b-bef926de3233" (UID: "5f9149be-f448-4a65-b69b-bef926de3233"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:13.173534 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.173503 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5j4t\" (UniqueName: \"kubernetes.io/projected/5f9149be-f448-4a65-b69b-bef926de3233-kube-api-access-c5j4t\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:13.173534 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.173532 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:13.173534 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.173541 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9149be-f448-4a65-b69b-bef926de3233-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:13.388962 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.388883 5736 generic.go:356] "Generic (PLEG): container finished" podID="5f9149be-f448-4a65-b69b-bef926de3233" containerID="ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a" exitCode=0 Oct 29 22:52:13.388962 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.388930 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" Oct 29 22:52:13.389480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.388963 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" event={"ID":"5f9149be-f448-4a65-b69b-bef926de3233","Type":"ContainerDied","Data":"ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a"} Oct 29 22:52:13.389480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.389014 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl" event={"ID":"5f9149be-f448-4a65-b69b-bef926de3233","Type":"ContainerDied","Data":"735032c104d09aa06cf22680aaee73ad27f7038bb96f8b5abd7b57cbd3efb1c3"} Oct 29 22:52:13.389480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.389036 5736 scope.go:122] "RemoveContainer" containerID="ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a" Oct 29 22:52:13.391170 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.391148 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" event={"ID":"ef78a262-e18e-4ec5-ae50-226bc71789a1","Type":"ContainerStarted","Data":"dbe537a4c8b7b3f860000466c566e5bed205f6ce5b6f83b9eb890d1b50f499fb"} Oct 29 22:52:13.404889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.404867 5736 scope.go:122] "RemoveContainer" containerID="ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a" Oct 29 22:52:13.405170 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:52:13.405152 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a\": container with ID starting with ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a not found: ID does not exist" containerID="ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a" Oct 29 22:52:13.405210 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.405177 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a"} err="failed to get container status \"ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a\": rpc error: code = NotFound desc = could not find container \"ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a\": container with ID starting with ba64f4f62c82db4ef0e479c9d08213b42e676c5bf00ac2770b621eb4800e9f4a not found: ID does not exist" Oct 29 22:52:13.485354 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.485319 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl"] Oct 29 22:52:13.489838 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.489806 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2bef9ca4-678d7fb466-4j2sl"] Oct 29 22:52:13.605994 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.605962 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:52:13.606174 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.606154 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 29 22:52:13.606214 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:13.606201 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 29 22:52:14.605568 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:14.605532 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 29 22:52:14.605977 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:14.605608 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 29 22:52:15.002617 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:15.002563 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9149be-f448-4a65-b69b-bef926de3233" path="/var/lib/kubelet/pods/5f9149be-f448-4a65-b69b-bef926de3233/volumes" Oct 29 22:52:15.606050 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:15.605800 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 29 22:52:15.606050 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:15.605855 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 29 22:52:16.605971 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:16.605931 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 29 22:52:16.606150 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:16.606004 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 29 22:52:17.606211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:17.606174 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 29 22:52:17.606722 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:17.606237 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 29 22:52:18.715751 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:18.715698 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:52:18.716408 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:18.716383 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:52:18.718157 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:18.718130 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 22:52:19.307440 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.307409 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv"] Oct 29 22:52:19.307879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.307860 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5f9149be-f448-4a65-b69b-bef926de3233" containerName="maas-api" Oct 29 22:52:19.307879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.307878 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9149be-f448-4a65-b69b-bef926de3233" containerName="maas-api" Oct 29 22:52:19.308051 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.307965 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9149be-f448-4a65-b69b-bef926de3233" containerName="maas-api" Oct 29 22:52:19.311103 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.311073 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.338108 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.338075 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"simulator-w0-e57216-kserve-self-signed-certs\"" Oct 29 22:52:19.338349 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.338326 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"default-dockercfg-6ltmm\"" Oct 29 22:52:19.338457 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.338356 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"openshift-service-ca.crt\"" Oct 29 22:52:19.338457 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.338404 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"kube-root-ca.crt\"" Oct 29 22:52:19.339977 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.339958 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv"] Oct 29 22:52:19.438571 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.438540 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-home\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.438749 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.438664 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/960f35db-200b-4e69-ae76-34f118f1af31-tls-certs\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.438749 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.438734 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-model-cache\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.438876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.438840 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwknv\" (UniqueName: \"kubernetes.io/projected/960f35db-200b-4e69-ae76-34f118f1af31-kube-api-access-vwknv\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.438934 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.438889 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-tmp-dir\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.439039 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.439013 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-dshm\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.539937 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.539900 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-home\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.540117 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.539963 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/960f35db-200b-4e69-ae76-34f118f1af31-tls-certs\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.540117 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.540003 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-model-cache\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.540117 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.540039 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwknv\" (UniqueName: \"kubernetes.io/projected/960f35db-200b-4e69-ae76-34f118f1af31-kube-api-access-vwknv\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.540117 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.540074 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-tmp-dir\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.540345 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.540132 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-dshm\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.540403 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.540356 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-home\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.540559 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.540530 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-tmp-dir\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.540695 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.540661 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-model-cache\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.543276 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.543251 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-dshm\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.543421 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.543400 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/960f35db-200b-4e69-ae76-34f118f1af31-tls-certs\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.566984 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.566913 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwknv\" (UniqueName: \"kubernetes.io/projected/960f35db-200b-4e69-ae76-34f118f1af31-kube-api-access-vwknv\") pod \"simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.624798 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.624766 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:19.921620 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:19.921508 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv"] Oct 29 22:52:19.925275 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:52:19.925247 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod960f35db_200b_4e69_ae76_34f118f1af31.slice/crio-7fe6dd603dd03bc18113200970f5741ce924b45c555c793093bc3ac5f8c8beeb WatchSource:0}: Error finding container 7fe6dd603dd03bc18113200970f5741ce924b45c555c793093bc3ac5f8c8beeb: Status 404 returned error can't find the container with id 7fe6dd603dd03bc18113200970f5741ce924b45c555c793093bc3ac5f8c8beeb Oct 29 22:52:20.428817 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:20.428777 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" event={"ID":"960f35db-200b-4e69-ae76-34f118f1af31","Type":"ContainerStarted","Data":"552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e"} Oct 29 22:52:20.429020 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:20.428827 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" event={"ID":"960f35db-200b-4e69-ae76-34f118f1af31","Type":"ContainerStarted","Data":"7fe6dd603dd03bc18113200970f5741ce924b45c555c793093bc3ac5f8c8beeb"} Oct 29 22:52:20.479719 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:20.479672 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podStartSLOduration=1.479659536 podStartE2EDuration="1.479659536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:20.47931184 +0000 UTC m=+537.595231446" watchObservedRunningTime="2026-10-29 22:52:20.479659536 +0000 UTC m=+537.595579142" Oct 29 22:52:20.625683 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:20.625648 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:21.009750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.009712 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2"] Oct 29 22:52:21.010223 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.009968 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" podUID="254ca785-8c0a-4288-a1e0-346e7e770822" containerName="maas-api" containerID="cri-o://82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf" gracePeriod=30 Oct 29 22:52:21.259366 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.259337 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:52:21.357877 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.357796 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-metrics-certs\") pod \"254ca785-8c0a-4288-a1e0-346e7e770822\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " Oct 29 22:52:21.357877 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.357864 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254ca785-8c0a-4288-a1e0-346e7e770822-kube-api-access-rvx24\" (UniqueName: \"kubernetes.io/projected/254ca785-8c0a-4288-a1e0-346e7e770822-kube-api-access-rvx24\") pod \"254ca785-8c0a-4288-a1e0-346e7e770822\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " Oct 29 22:52:21.358111 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.357928 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls\") pod \"254ca785-8c0a-4288-a1e0-346e7e770822\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " Oct 29 22:52:21.358111 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.357996 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254ca785-8c0a-4288-a1e0-346e7e770822-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ca785-8c0a-4288-a1e0-346e7e770822-openshift-service-ca\") pod \"254ca785-8c0a-4288-a1e0-346e7e770822\" (UID: \"254ca785-8c0a-4288-a1e0-346e7e770822\") " Oct 29 22:52:21.358376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.358348 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254ca785-8c0a-4288-a1e0-346e7e770822-openshift-service-ca" pod "254ca785-8c0a-4288-a1e0-346e7e770822" (UID: "254ca785-8c0a-4288-a1e0-346e7e770822"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:21.359965 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.359941 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls" pod "254ca785-8c0a-4288-a1e0-346e7e770822" (UID: "254ca785-8c0a-4288-a1e0-346e7e770822"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:21.360063 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.359998 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-metrics-certs" pod "254ca785-8c0a-4288-a1e0-346e7e770822" (UID: "254ca785-8c0a-4288-a1e0-346e7e770822"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:21.360121 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.360075 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254ca785-8c0a-4288-a1e0-346e7e770822-kube-api-access-rvx24" pod "254ca785-8c0a-4288-a1e0-346e7e770822" (UID: "254ca785-8c0a-4288-a1e0-346e7e770822"). InnerVolumeSpecName "kube-api-access-rvx24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:21.437558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.437519 5736 generic.go:356] "Generic (PLEG): container finished" podID="254ca785-8c0a-4288-a1e0-346e7e770822" containerID="82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf" exitCode=0 Oct 29 22:52:21.438425 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.438307 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" Oct 29 22:52:21.443688 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.443658 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" event={"ID":"254ca785-8c0a-4288-a1e0-346e7e770822","Type":"ContainerDied","Data":"82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf"} Oct 29 22:52:21.443818 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.443727 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2" event={"ID":"254ca785-8c0a-4288-a1e0-346e7e770822","Type":"ContainerDied","Data":"868d6d3026a0a34d565da909f382049f1294009bb28c62eefd5032899b69b19b"} Oct 29 22:52:21.443818 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.443753 5736 scope.go:122] "RemoveContainer" containerID="82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf" Oct 29 22:52:21.454509 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.454486 5736 scope.go:122] "RemoveContainer" containerID="82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf" Oct 29 22:52:21.454877 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:52:21.454839 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf\": container with ID starting with 82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf not found: ID does not exist" containerID="82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf" Oct 29 22:52:21.454986 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.454888 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf"} err="failed to get container status \"82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf\": rpc error: code = NotFound desc = could not find container \"82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf\": container with ID starting with 82f9447a4311a39e90c7e237a30a605f4a3ee31cd9391ed39373fbb944eb8dcf not found: ID does not exist" Oct 29 22:52:21.459233 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.459210 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.459319 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.459240 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254ca785-8c0a-4288-a1e0-346e7e770822-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.459319 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.459259 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254ca785-8c0a-4288-a1e0-346e7e770822-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.459319 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.459274 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvx24\" (UniqueName: \"kubernetes.io/projected/254ca785-8c0a-4288-a1e0-346e7e770822-kube-api-access-rvx24\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.490150 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.490125 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2"] Oct 29 22:52:21.494852 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:21.494828 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5db969e0-5db8d9777f-ntpd2"] Oct 29 22:52:23.004758 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:23.004720 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254ca785-8c0a-4288-a1e0-346e7e770822" path="/var/lib/kubelet/pods/254ca785-8c0a-4288-a1e0-346e7e770822/volumes" Oct 29 22:52:25.165051 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.165015 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6"] Oct 29 22:52:25.165442 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.165273 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" podUID="8f292623-0627-41f2-974a-239ff5b3e739" containerName="maas-api" containerID="cri-o://395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722" gracePeriod=30 Oct 29 22:52:25.416009 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.415955 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:52:25.461253 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.461225 5736 generic.go:356] "Generic (PLEG): container finished" podID="8f292623-0627-41f2-974a-239ff5b3e739" containerID="395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722" exitCode=0 Oct 29 22:52:25.461410 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.461270 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" event={"ID":"8f292623-0627-41f2-974a-239ff5b3e739","Type":"ContainerDied","Data":"395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722"} Oct 29 22:52:25.461410 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.461276 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" Oct 29 22:52:25.461410 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.461315 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6" event={"ID":"8f292623-0627-41f2-974a-239ff5b3e739","Type":"ContainerDied","Data":"2c9ff8936149847238f443522fcc9e8c959f485b59d402f553ae5fb124db88df"} Oct 29 22:52:25.461410 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.461337 5736 scope.go:122] "RemoveContainer" containerID="395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722" Oct 29 22:52:25.471543 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.471515 5736 scope.go:122] "RemoveContainer" containerID="395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722" Oct 29 22:52:25.471884 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:52:25.471861 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722\": container with ID starting with 395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722 not found: ID does not exist" containerID="395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722" Oct 29 22:52:25.471982 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.471894 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722"} err="failed to get container status \"395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722\": rpc error: code = NotFound desc = could not find container \"395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722\": container with ID starting with 395e5fff91a212f407ea8f9bf95dcd3c055203563cc04f6e07956188cf23c722 not found: ID does not exist" Oct 29 22:52:25.496629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.496606 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f292623-0627-41f2-974a-239ff5b3e739-kube-api-access-mfkct\" (UniqueName: \"kubernetes.io/projected/8f292623-0627-41f2-974a-239ff5b3e739-kube-api-access-mfkct\") pod \"8f292623-0627-41f2-974a-239ff5b3e739\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " Oct 29 22:52:25.496750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.496645 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f292623-0627-41f2-974a-239ff5b3e739-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f292623-0627-41f2-974a-239ff5b3e739-openshift-service-ca\") pod \"8f292623-0627-41f2-974a-239ff5b3e739\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " Oct 29 22:52:25.496750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.496667 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-metrics-certs\") pod \"8f292623-0627-41f2-974a-239ff5b3e739\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " Oct 29 22:52:25.496750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.496709 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls\") pod \"8f292623-0627-41f2-974a-239ff5b3e739\" (UID: \"8f292623-0627-41f2-974a-239ff5b3e739\") " Oct 29 22:52:25.496999 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.496972 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f292623-0627-41f2-974a-239ff5b3e739-openshift-service-ca" pod "8f292623-0627-41f2-974a-239ff5b3e739" (UID: "8f292623-0627-41f2-974a-239ff5b3e739"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:25.498691 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.498663 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f292623-0627-41f2-974a-239ff5b3e739-kube-api-access-mfkct" pod "8f292623-0627-41f2-974a-239ff5b3e739" (UID: "8f292623-0627-41f2-974a-239ff5b3e739"). InnerVolumeSpecName "kube-api-access-mfkct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:25.498818 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.498794 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-metrics-certs" pod "8f292623-0627-41f2-974a-239ff5b3e739" (UID: "8f292623-0627-41f2-974a-239ff5b3e739"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:25.498888 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.498871 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls" pod "8f292623-0627-41f2-974a-239ff5b3e739" (UID: "8f292623-0627-41f2-974a-239ff5b3e739"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:25.598456 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.598421 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfkct\" (UniqueName: \"kubernetes.io/projected/8f292623-0627-41f2-974a-239ff5b3e739-kube-api-access-mfkct\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:25.598456 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.598456 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f292623-0627-41f2-974a-239ff5b3e739-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:25.598456 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.598470 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:25.598748 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.598483 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f292623-0627-41f2-974a-239ff5b3e739-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:25.805793 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.805765 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6"] Oct 29 22:52:25.819606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:25.818139 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e406071d-779d879f97-wvxn6"] Oct 29 22:52:26.823349 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:52:26.823306 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:52:27.002404 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:27.002367 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f292623-0627-41f2-974a-239ff5b3e739" path="/var/lib/kubelet/pods/8f292623-0627-41f2-974a-239ff5b3e739/volumes" Oct 29 22:52:28.348434 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.348403 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg"] Oct 29 22:52:28.348879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.348853 5736 cpu_manager.go:414] "containerMap: removing container" podUID="254ca785-8c0a-4288-a1e0-346e7e770822" containerName="maas-api" Oct 29 22:52:28.348879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.348869 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254ca785-8c0a-4288-a1e0-346e7e770822" containerName="maas-api" Oct 29 22:52:28.348987 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.348889 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8f292623-0627-41f2-974a-239ff5b3e739" containerName="maas-api" Oct 29 22:52:28.348987 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.348898 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f292623-0627-41f2-974a-239ff5b3e739" containerName="maas-api" Oct 29 22:52:28.349087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.349009 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="254ca785-8c0a-4288-a1e0-346e7e770822" containerName="maas-api" Oct 29 22:52:28.349087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.349023 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f292623-0627-41f2-974a-239ff5b3e739" containerName="maas-api" Oct 29 22:52:28.352203 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.352179 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.380192 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.380171 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"simulator-w1-99dfb0-kserve-self-signed-certs\"" Oct 29 22:52:28.380433 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.380414 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"kube-root-ca.crt\"" Oct 29 22:52:28.380493 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.380457 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"openshift-service-ca.crt\"" Oct 29 22:52:28.380615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.380567 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"default-dockercfg-nf4k7\"" Oct 29 22:52:28.382264 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.382239 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg"] Oct 29 22:52:28.427119 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.427087 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-home\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.427240 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.427132 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-tmp-dir\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.427281 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.427225 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-model-cache\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.427340 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.427277 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvc2\" (UniqueName: \"kubernetes.io/projected/674b738e-397f-4f31-991b-93fdcabf6451-kube-api-access-qvvc2\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.427429 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.427409 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-dshm\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.427591 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.427561 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/674b738e-397f-4f31-991b-93fdcabf6451-tls-certs\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.528622 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.528567 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-home\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.528771 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.528632 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-tmp-dir\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.528771 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.528662 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-model-cache\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.528771 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.528692 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvc2\" (UniqueName: \"kubernetes.io/projected/674b738e-397f-4f31-991b-93fdcabf6451-kube-api-access-qvvc2\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.528771 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.528760 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-dshm\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.529012 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.528876 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/674b738e-397f-4f31-991b-93fdcabf6451-tls-certs\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.529078 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.529007 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-home\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.529078 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.529029 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-tmp-dir\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.529191 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.529109 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-model-cache\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.531136 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.531104 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-dshm\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.531347 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.531328 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/674b738e-397f-4f31-991b-93fdcabf6451-tls-certs\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.550071 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.550046 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvc2\" (UniqueName: \"kubernetes.io/projected/674b738e-397f-4f31-991b-93fdcabf6451-kube-api-access-qvvc2\") pod \"simulator-w1-99dfb0-kserve-677984f898-dwdwg\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.663769 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.663669 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:28.960303 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:28.960209 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg"] Oct 29 22:52:28.964452 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:52:28.964424 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod674b738e_397f_4f31_991b_93fdcabf6451.slice/crio-d88051e83818535542e95a5cc6d9d39105c64a280ac1854782e4af00779db85b WatchSource:0}: Error finding container d88051e83818535542e95a5cc6d9d39105c64a280ac1854782e4af00779db85b: Status 404 returned error can't find the container with id d88051e83818535542e95a5cc6d9d39105c64a280ac1854782e4af00779db85b Oct 29 22:52:29.485062 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:29.485022 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" event={"ID":"674b738e-397f-4f31-991b-93fdcabf6451","Type":"ContainerStarted","Data":"3497f57649ff489e0478acd7bd1d7b078f67ab1cf7e4313fda9f68a409d33491"} Oct 29 22:52:29.485623 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:29.485085 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" event={"ID":"674b738e-397f-4f31-991b-93fdcabf6451","Type":"ContainerStarted","Data":"d88051e83818535542e95a5cc6d9d39105c64a280ac1854782e4af00779db85b"} Oct 29 22:52:29.523542 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:29.523492 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podStartSLOduration=1.523480368 podStartE2EDuration="1.523480368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:29.522453277 +0000 UTC m=+546.638372893" watchObservedRunningTime="2026-10-29 22:52:29.523480368 +0000 UTC m=+546.639399974" Oct 29 22:52:29.625957 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:29.625924 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:29.639528 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:29.639503 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:29.664513 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:29.664482 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:30.500939 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:30.500911 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:52:31.603504 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.603470 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8"] Oct 29 22:52:31.604016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.603731 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" podUID="2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" containerName="maas-api" containerID="cri-o://778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb" gracePeriod=30 Oct 29 22:52:31.842018 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.841993 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:52:31.964514 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.964489 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-kube-api-access-sfwjq\" (UniqueName: \"kubernetes.io/projected/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-kube-api-access-sfwjq\") pod \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " Oct 29 22:52:31.964705 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.964537 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-maas-api-tls\") pod \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " Oct 29 22:52:31.964705 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.964564 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-metrics-certs\") pod \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " Oct 29 22:52:31.964799 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.964710 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-openshift-service-ca\") pod \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\" (UID: \"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c\") " Oct 29 22:52:31.965133 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.965107 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-openshift-service-ca" pod "2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" (UID: "2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:31.966818 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.966799 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-maas-api-tls" pod "2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" (UID: "2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:31.967160 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.967136 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-metrics-certs" pod "2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" (UID: "2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:31.967229 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:31.967173 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-kube-api-access-sfwjq" pod "2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" (UID: "2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c"). InnerVolumeSpecName "kube-api-access-sfwjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:32.066388 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.066358 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:32.066388 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.066389 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:32.066573 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.066404 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:32.066573 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.066419 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfwjq\" (UniqueName: \"kubernetes.io/projected/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c-kube-api-access-sfwjq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:32.077610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.077556 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn"] Oct 29 22:52:32.077830 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.077806 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" podUID="8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" containerName="maas-api" containerID="cri-o://78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85" gracePeriod=30 Oct 29 22:52:32.316389 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.316365 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:52:32.469609 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.469559 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-metrics-certs\") pod \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " Oct 29 22:52:32.469811 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.469730 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-maas-api-tls\") pod \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " Oct 29 22:52:32.469811 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.469775 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-kube-api-access-hkmcw\" (UniqueName: \"kubernetes.io/projected/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-kube-api-access-hkmcw\") pod \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " Oct 29 22:52:32.469811 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.469805 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-openshift-service-ca\") pod \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\" (UID: \"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0\") " Oct 29 22:52:32.470781 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.470438 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-openshift-service-ca" pod "8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" (UID: "8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:32.472048 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.472023 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-metrics-certs" pod "8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" (UID: "8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:32.472909 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.472884 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-maas-api-tls" pod "8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" (UID: "8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:32.473129 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.473108 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-kube-api-access-hkmcw" pod "8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" (UID: "8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0"). InnerVolumeSpecName "kube-api-access-hkmcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:32.504645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.503133 5736 generic.go:356] "Generic (PLEG): container finished" podID="2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" containerID="778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb" exitCode=0 Oct 29 22:52:32.504645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.503429 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" event={"ID":"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c","Type":"ContainerDied","Data":"778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb"} Oct 29 22:52:32.504645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.503543 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" event={"ID":"2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c","Type":"ContainerDied","Data":"039ada2dc613a9e17decd2b4afc9d124a8fca2cd3b30b0d6d7d7dcf2c576ec9b"} Oct 29 22:52:32.504645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.503595 5736 scope.go:122] "RemoveContainer" containerID="778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb" Oct 29 22:52:32.504645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.503974 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8" Oct 29 22:52:32.525760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.525126 5736 generic.go:356] "Generic (PLEG): container finished" podID="8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" containerID="78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85" exitCode=0 Oct 29 22:52:32.525760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.525193 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" event={"ID":"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0","Type":"ContainerDied","Data":"78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85"} Oct 29 22:52:32.525760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.525239 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" event={"ID":"8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0","Type":"ContainerDied","Data":"da1cfdb815e70fa9673da60a7b916f64a98b26a3633d6a4fbba74cf38935a6f6"} Oct 29 22:52:32.525760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.525343 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn" Oct 29 22:52:32.531719 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.531693 5736 scope.go:122] "RemoveContainer" containerID="778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb" Oct 29 22:52:32.532049 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:52:32.532027 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb\": container with ID starting with 778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb not found: ID does not exist" containerID="778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb" Oct 29 22:52:32.532143 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.532057 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb"} err="failed to get container status \"778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb\": rpc error: code = NotFound desc = could not find container \"778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb\": container with ID starting with 778038c1acdaf9a0fc0455a3cf1aaf2c172362e4531dbe3507d99690d18f9abb not found: ID does not exist" Oct 29 22:52:32.532143 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.532075 5736 scope.go:122] "RemoveContainer" containerID="78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85" Oct 29 22:52:32.541364 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.541277 5736 scope.go:122] "RemoveContainer" containerID="78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85" Oct 29 22:52:32.541630 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:52:32.541610 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85\": container with ID starting with 78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85 not found: ID does not exist" containerID="78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85" Oct 29 22:52:32.541709 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.541643 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85"} err="failed to get container status \"78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85\": rpc error: code = NotFound desc = could not find container \"78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85\": container with ID starting with 78e28864e85cc03afc2edca754a54e82883657deafce5a1c94387d509f84af85 not found: ID does not exist" Oct 29 22:52:32.560942 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.560914 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8"] Oct 29 22:52:32.566064 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.566038 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-865d946f-77cb5ff8fc-kxwh8"] Oct 29 22:52:32.571011 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.570988 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:32.571086 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.571014 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkmcw\" (UniqueName: \"kubernetes.io/projected/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-kube-api-access-hkmcw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:32.571086 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.571024 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:32.571086 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.571034 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:52:32.612346 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.612321 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn"] Oct 29 22:52:32.616727 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:32.616701 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8042816e-6f86f79447-lcxwn"] Oct 29 22:52:33.002429 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.002396 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" path="/var/lib/kubelet/pods/2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c/volumes" Oct 29 22:52:33.002820 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.002807 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" path="/var/lib/kubelet/pods/8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0/volumes" Oct 29 22:52:33.825549 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.825455 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch"] Oct 29 22:52:33.826148 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.826114 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" containerName="maas-api" Oct 29 22:52:33.826148 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.826141 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" containerName="maas-api" Oct 29 22:52:33.826395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.826159 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" containerName="maas-api" Oct 29 22:52:33.826395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.826168 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" containerName="maas-api" Oct 29 22:52:33.826395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.826308 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f51d2ba-5e4f-40fb-bc3c-f21edbcfcc3c" containerName="maas-api" Oct 29 22:52:33.826395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.826321 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb19ec2-d9ee-4c3c-9525-4f758c09a5e0" containerName="maas-api" Oct 29 22:52:33.829630 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.829606 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:33.858604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.858558 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch"] Oct 29 22:52:33.882272 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.882243 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l"] Oct 29 22:52:33.884999 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.884979 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:33.897494 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.897469 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l"] Oct 29 22:52:33.909995 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.909967 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-1ee571\"" Oct 29 22:52:33.910112 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.909996 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7aeaad-openshift-default-dockercfg-bxl84\"" Oct 29 22:52:33.910247 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.910228 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-1ee571-dockercfg-nz2br\"" Oct 29 22:52:33.985597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985160 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:33.985597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985281 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cfaf69-db9f-4bee-ac82-b012b725172d-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:33.985597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985320 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:33.985597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985341 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:33.985597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985426 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:33.985597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985563 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:33.985911 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985638 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbkbr\" (UniqueName: \"kubernetes.io/projected/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-kube-api-access-pbkbr\") pod \"payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l\" (UID: \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:33.985911 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985680 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmnn\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-kube-api-access-fkmnn\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:33.985911 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985710 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:33.985911 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985749 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l\" (UID: \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:33.985911 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:33.985770 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.086694 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.086606 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.086862 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.086699 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cfaf69-db9f-4bee-ac82-b012b725172d-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.086862 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.086735 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.086991 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.086857 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.086991 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.086902 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.086991 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.086942 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.086991 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.086977 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.087189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.086991 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbkbr\" (UniqueName: \"kubernetes.io/projected/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-kube-api-access-pbkbr\") pod \"payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l\" (UID: \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:34.087189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.087026 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmnn\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-kube-api-access-fkmnn\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.087189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.087054 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.087189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.087087 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l\" (UID: \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:34.087189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.087114 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.087432 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.087300 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.087432 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.087310 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.087541 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.087438 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cfaf69-db9f-4bee-ac82-b012b725172d-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.087541 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.087453 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.088055 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.088031 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l\" (UID: \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:34.089680 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.089656 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.089899 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.089878 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.108439 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.108418 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.108517 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.108441 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmnn\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-kube-api-access-fkmnn\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.108517 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.108454 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbkbr\" (UniqueName: \"kubernetes.io/projected/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-kube-api-access-pbkbr\") pod \"payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l\" (UID: \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:34.144528 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.144490 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:34.197219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.196786 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:34.461525 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.461490 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch"] Oct 29 22:52:34.465491 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:52:34.465462 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cfaf69_db9f_4bee_ac82_b012b725172d.slice/crio-101f0d14a92d2634872a3f94c491ae0fbf511900fc66e6dfb4b27f32ffbdd072 WatchSource:0}: Error finding container 101f0d14a92d2634872a3f94c491ae0fbf511900fc66e6dfb4b27f32ffbdd072: Status 404 returned error can't find the container with id 101f0d14a92d2634872a3f94c491ae0fbf511900fc66e6dfb4b27f32ffbdd072 Oct 29 22:52:34.467780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.467747 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:52:34.467850 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.467823 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:52:34.467892 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.467861 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:52:34.506016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.505990 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l"] Oct 29 22:52:34.510345 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:52:34.510317 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1632fbd9_04f1_4a5b_a40f_e778a40a3e55.slice/crio-20baac6d3f10f66cee7bcb68d505324001ca9a2e98c818641a2f1fe34a4b1584 WatchSource:0}: Error finding container 20baac6d3f10f66cee7bcb68d505324001ca9a2e98c818641a2f1fe34a4b1584: Status 404 returned error can't find the container with id 20baac6d3f10f66cee7bcb68d505324001ca9a2e98c818641a2f1fe34a4b1584 Oct 29 22:52:34.554834 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.554798 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" event={"ID":"54cfaf69-db9f-4bee-ac82-b012b725172d","Type":"ContainerStarted","Data":"421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06"} Oct 29 22:52:34.555310 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.555285 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" event={"ID":"54cfaf69-db9f-4bee-ac82-b012b725172d","Type":"ContainerStarted","Data":"101f0d14a92d2634872a3f94c491ae0fbf511900fc66e6dfb4b27f32ffbdd072"} Oct 29 22:52:34.569026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.560487 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" event={"ID":"1632fbd9-04f1-4a5b-a40f-e778a40a3e55","Type":"ContainerStarted","Data":"20baac6d3f10f66cee7bcb68d505324001ca9a2e98c818641a2f1fe34a4b1584"} Oct 29 22:52:34.615420 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:34.614956 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" podStartSLOduration=1.614909095 podStartE2EDuration="1.614909095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:34.614054458 +0000 UTC m=+551.729974064" watchObservedRunningTime="2026-10-29 22:52:34.614909095 +0000 UTC m=+551.730828703" Oct 29 22:52:35.144911 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:35.144875 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:35.151053 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:35.151029 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:35.566207 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:35.566176 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:35.567187 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:35.567164 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:52:37.576833 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:37.576792 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" event={"ID":"1632fbd9-04f1-4a5b-a40f-e778a40a3e55","Type":"ContainerStarted","Data":"13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46"} Oct 29 22:52:37.577341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:37.576948 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:37.613954 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:37.613913 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" podStartSLOduration=2.522828696 podStartE2EDuration="4.613900384s" totalImagesPullingTime="2.091071688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:37.613143932 +0000 UTC m=+554.729063541" watchObservedRunningTime="2026-10-29 22:52:37.613900384 +0000 UTC m=+554.729819968" Oct 29 22:52:38.664506 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:38.664470 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:38.676547 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:38.676523 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:39.137876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.137838 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd"] Oct 29 22:52:39.141802 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.141781 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:39.158196 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.158134 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-7aeaad\"" Oct 29 22:52:39.158353 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.158321 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-7aeaad-dockercfg-4l84k\"" Oct 29 22:52:39.167810 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.167735 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd"] Oct 29 22:52:39.239921 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.239891 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e21279-18ed-4bc8-9214-f93a3c416a36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd\" (UID: \"d8e21279-18ed-4bc8-9214-f93a3c416a36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:39.240068 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.239954 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qg77\" (UniqueName: \"kubernetes.io/projected/d8e21279-18ed-4bc8-9214-f93a3c416a36-kube-api-access-5qg77\") pod \"payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd\" (UID: \"d8e21279-18ed-4bc8-9214-f93a3c416a36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:39.341320 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.341285 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qg77\" (UniqueName: \"kubernetes.io/projected/d8e21279-18ed-4bc8-9214-f93a3c416a36-kube-api-access-5qg77\") pod \"payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd\" (UID: \"d8e21279-18ed-4bc8-9214-f93a3c416a36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:39.341465 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.341403 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e21279-18ed-4bc8-9214-f93a3c416a36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd\" (UID: \"d8e21279-18ed-4bc8-9214-f93a3c416a36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:39.342130 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.342110 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e21279-18ed-4bc8-9214-f93a3c416a36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd\" (UID: \"d8e21279-18ed-4bc8-9214-f93a3c416a36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:39.363229 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.363199 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qg77\" (UniqueName: \"kubernetes.io/projected/d8e21279-18ed-4bc8-9214-f93a3c416a36-kube-api-access-5qg77\") pod \"payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd\" (UID: \"d8e21279-18ed-4bc8-9214-f93a3c416a36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:39.458745 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.458681 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:39.597496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.597470 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:52:39.759767 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:39.759722 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd"] Oct 29 22:52:39.764393 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:52:39.764367 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e21279_18ed_4bc8_9214_f93a3c416a36.slice/crio-db2aa42ce93462a5a9c29fa2fa778a37adc79230e50fca74cb0553094b799ff1 WatchSource:0}: Error finding container db2aa42ce93462a5a9c29fa2fa778a37adc79230e50fca74cb0553094b799ff1: Status 404 returned error can't find the container with id db2aa42ce93462a5a9c29fa2fa778a37adc79230e50fca74cb0553094b799ff1 Oct 29 22:52:40.593876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:40.593821 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" event={"ID":"d8e21279-18ed-4bc8-9214-f93a3c416a36","Type":"ContainerStarted","Data":"df52c9d7919fda9ce85014e2f2059de10ebb07719fade8e455fc3aa4dcab6bed"} Oct 29 22:52:40.594082 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:40.593899 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" event={"ID":"d8e21279-18ed-4bc8-9214-f93a3c416a36","Type":"ContainerStarted","Data":"db2aa42ce93462a5a9c29fa2fa778a37adc79230e50fca74cb0553094b799ff1"} Oct 29 22:52:40.594082 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:40.593999 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:40.631955 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:40.631870 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" podStartSLOduration=1.631854831 podStartE2EDuration="1.631854831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:40.630508462 +0000 UTC m=+557.746428067" watchObservedRunningTime="2026-10-29 22:52:40.631854831 +0000 UTC m=+557.747774440" Oct 29 22:52:43.015598 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.015554 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww"] Oct 29 22:52:43.019431 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.019407 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.035871 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.035847 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"premium-w0-e57216-kserve-self-signed-certs\"" Oct 29 22:52:43.047440 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.047409 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww"] Oct 29 22:52:43.179033 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.179003 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd831ca-e923-49fb-9993-fb6954bf068b-tls-certs\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.179208 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.179056 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-model-cache\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.179246 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.179194 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxd9v\" (UniqueName: \"kubernetes.io/projected/fcd831ca-e923-49fb-9993-fb6954bf068b-kube-api-access-xxd9v\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.179290 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.179252 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-home\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.179290 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.179270 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-dshm\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.179378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.179355 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-tmp-dir\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.280087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.279998 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxd9v\" (UniqueName: \"kubernetes.io/projected/fcd831ca-e923-49fb-9993-fb6954bf068b-kube-api-access-xxd9v\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.280087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.280058 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-home\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.280087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.280080 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-dshm\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.280294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.280106 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-tmp-dir\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.280294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.280146 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd831ca-e923-49fb-9993-fb6954bf068b-tls-certs\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.280294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.280173 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-model-cache\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.280455 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.280433 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-home\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.280566 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.280542 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-model-cache\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.280652 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.280616 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-tmp-dir\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.282641 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.282617 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-dshm\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.282975 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.282953 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd831ca-e923-49fb-9993-fb6954bf068b-tls-certs\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.301317 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.301295 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxd9v\" (UniqueName: \"kubernetes.io/projected/fcd831ca-e923-49fb-9993-fb6954bf068b-kube-api-access-xxd9v\") pod \"premium-w0-e57216-kserve-6b9b9574c4-tbhww\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.334197 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.334171 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:43.632331 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:43.632200 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww"] Oct 29 22:52:43.636238 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:52:43.636195 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcd831ca_e923_49fb_9993_fb6954bf068b.slice/crio-df75d5a2df047df7eba59ba7733d63dc13194637711d3abbbd21e557587038bf WatchSource:0}: Error finding container df75d5a2df047df7eba59ba7733d63dc13194637711d3abbbd21e557587038bf: Status 404 returned error can't find the container with id df75d5a2df047df7eba59ba7733d63dc13194637711d3abbbd21e557587038bf Oct 29 22:52:44.614898 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:44.614856 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" event={"ID":"fcd831ca-e923-49fb-9993-fb6954bf068b","Type":"ContainerStarted","Data":"60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709"} Oct 29 22:52:44.614898 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:44.614900 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" event={"ID":"fcd831ca-e923-49fb-9993-fb6954bf068b","Type":"ContainerStarted","Data":"df75d5a2df047df7eba59ba7733d63dc13194637711d3abbbd21e557587038bf"} Oct 29 22:52:44.654239 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:44.654187 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podStartSLOduration=2.654172521 podStartE2EDuration="2.654172521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:44.652734581 +0000 UTC m=+561.768654187" watchObservedRunningTime="2026-10-29 22:52:44.654172521 +0000 UTC m=+561.770092126" Oct 29 22:52:45.334848 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:45.334814 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:48.585053 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:48.585024 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 22:52:51.601849 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:51.601818 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 22:52:53.334879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:53.334846 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:53.347892 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:53.347867 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:53.672677 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:53.672575 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:52:54.911931 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.911897 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn"] Oct 29 22:52:54.919876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.919854 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:54.942050 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.941980 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn"] Oct 29 22:52:54.943029 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.942997 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"simulator-w3-32c192-kserve-self-signed-certs\"" Oct 29 22:52:54.955143 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.945750 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"openshift-service-ca.crt\"" Oct 29 22:52:54.955827 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.945835 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"kube-root-ca.crt\"" Oct 29 22:52:54.956012 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.945920 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"default-dockercfg-rl87m\"" Oct 29 22:52:54.991061 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.991022 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-home\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:54.991247 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.991146 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-dshm\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:54.991247 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.991210 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-model-cache\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:54.991331 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.991258 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-tmp-dir\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:54.991370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.991340 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzk7\" (UniqueName: \"kubernetes.io/projected/902df38e-ddc0-457e-8877-2702115f3de8-kube-api-access-xzzk7\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:54.991404 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:54.991393 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902df38e-ddc0-457e-8877-2702115f3de8-tls-certs\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.092047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.092001 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-home\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.092214 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.092089 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-dshm\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.092214 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.092128 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-model-cache\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.092214 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.092152 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-tmp-dir\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.092401 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.092247 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzk7\" (UniqueName: \"kubernetes.io/projected/902df38e-ddc0-457e-8877-2702115f3de8-kube-api-access-xzzk7\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.092401 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.092391 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902df38e-ddc0-457e-8877-2702115f3de8-tls-certs\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.092511 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.092454 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-home\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.092850 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.092822 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-tmp-dir\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.093364 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.093179 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-model-cache\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.095145 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.095116 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-dshm\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.095760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.095735 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902df38e-ddc0-457e-8877-2702115f3de8-tls-certs\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.114063 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.114036 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzk7\" (UniqueName: \"kubernetes.io/projected/902df38e-ddc0-457e-8877-2702115f3de8-kube-api-access-xzzk7\") pod \"simulator-w3-32c192-kserve-85c885c48d-gxkrn\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.243604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.236792 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:55.565958 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.565871 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn"] Oct 29 22:52:55.569948 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:52:55.569920 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902df38e_ddc0_457e_8877_2702115f3de8.slice/crio-c2d96713c0a4231241dee6530086329f82e8edd769c1f25e88200116cfac1003 WatchSource:0}: Error finding container c2d96713c0a4231241dee6530086329f82e8edd769c1f25e88200116cfac1003: Status 404 returned error can't find the container with id c2d96713c0a4231241dee6530086329f82e8edd769c1f25e88200116cfac1003 Oct 29 22:52:55.679790 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.679755 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" event={"ID":"902df38e-ddc0-457e-8877-2702115f3de8","Type":"ContainerStarted","Data":"ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6"} Oct 29 22:52:55.679979 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.679798 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" event={"ID":"902df38e-ddc0-457e-8877-2702115f3de8","Type":"ContainerStarted","Data":"c2d96713c0a4231241dee6530086329f82e8edd769c1f25e88200116cfac1003"} Oct 29 22:52:55.720282 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:55.720222 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podStartSLOduration=1.720202153 podStartE2EDuration="1.720202153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:55.71754043 +0000 UTC m=+572.833460035" watchObservedRunningTime="2026-10-29 22:52:55.720202153 +0000 UTC m=+572.836121759" Oct 29 22:52:56.247604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:56.242656 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:52:56.822666 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:52:56.822631 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:52:57.545111 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.545074 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf"] Oct 29 22:52:57.553931 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.553905 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.567372 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.567342 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf"] Oct 29 22:52:57.575635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.575575 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"premium-w1-99dfb0-kserve-self-signed-certs\"" Oct 29 22:52:57.616441 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.616403 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-dshm\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.616441 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.616437 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-tmp-dir\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.616683 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.616458 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f33759-614b-4b2c-923f-1c7d238bbb8d-tls-certs\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.616683 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.616628 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5fnj\" (UniqueName: \"kubernetes.io/projected/06f33759-614b-4b2c-923f-1c7d238bbb8d-kube-api-access-p5fnj\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.616791 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.616702 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-home\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.616791 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.616751 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-model-cache\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.717769 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.717730 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-home\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.717769 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.717774 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-model-cache\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.718019 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.717876 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-dshm\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.718019 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.717904 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-tmp-dir\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.718019 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.717932 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f33759-614b-4b2c-923f-1c7d238bbb8d-tls-certs\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.718019 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.717979 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5fnj\" (UniqueName: \"kubernetes.io/projected/06f33759-614b-4b2c-923f-1c7d238bbb8d-kube-api-access-p5fnj\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.718239 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.718160 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-home\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.718239 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.718218 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-model-cache\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.718753 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.718723 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-tmp-dir\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.720445 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.720421 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-dshm\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.720605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.720518 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f33759-614b-4b2c-923f-1c7d238bbb8d-tls-certs\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.741267 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.741206 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5fnj\" (UniqueName: \"kubernetes.io/projected/06f33759-614b-4b2c-923f-1c7d238bbb8d-kube-api-access-p5fnj\") pod \"premium-w1-99dfb0-kserve-795bb9db5f-bp5xf\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:57.878947 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:57.878857 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:52:58.185516 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:58.185432 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf"] Oct 29 22:52:58.189257 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:52:58.189219 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f33759_614b_4b2c_923f_1c7d238bbb8d.slice/crio-9a6da5d1a20ac3261727827bbb54678ada69248c391a5be0f0ddde81da0127b1 WatchSource:0}: Error finding container 9a6da5d1a20ac3261727827bbb54678ada69248c391a5be0f0ddde81da0127b1: Status 404 returned error can't find the container with id 9a6da5d1a20ac3261727827bbb54678ada69248c391a5be0f0ddde81da0127b1 Oct 29 22:52:58.694803 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:58.694761 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" event={"ID":"06f33759-614b-4b2c-923f-1c7d238bbb8d","Type":"ContainerStarted","Data":"4a2dc493cf8eb0a7e93d227cb1f64eae1e17ebb3eb8d2301f89bd08eecc16dd3"} Oct 29 22:52:58.695170 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:58.694810 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" event={"ID":"06f33759-614b-4b2c-923f-1c7d238bbb8d","Type":"ContainerStarted","Data":"9a6da5d1a20ac3261727827bbb54678ada69248c391a5be0f0ddde81da0127b1"} Oct 29 22:52:58.879318 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:52:58.879286 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:53:03.722752 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:03.722721 5736 generic.go:356] "Generic (PLEG): container finished" podID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerID="421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06" exitCode=0 Oct 29 22:53:03.723221 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:03.722793 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" event={"ID":"54cfaf69-db9f-4bee-ac82-b012b725172d","Type":"ContainerDied","Data":"421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06"} Oct 29 22:53:03.723221 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:03.723155 5736 scope.go:122] "RemoveContainer" containerID="421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06" Oct 29 22:53:03.724387 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:03.724356 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:03.724485 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:03.724438 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:03.724546 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:03.724482 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:03.783650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:03.778058 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podStartSLOduration=6.778040758 podStartE2EDuration="6.778040758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:58.748782678 +0000 UTC m=+575.864702285" watchObservedRunningTime="2026-10-29 22:53:03.778040758 +0000 UTC m=+580.893960365" Oct 29 22:53:04.730991 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:04.730942 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" event={"ID":"54cfaf69-db9f-4bee-ac82-b012b725172d","Type":"ContainerStarted","Data":"4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460"} Oct 29 22:53:05.146664 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:05.145928 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:53:05.151603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:05.149692 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 29 22:53:05.151603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:05.149765 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 29 22:53:05.243602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:05.243403 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:53:05.268115 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:05.268086 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:53:05.566433 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:05.566403 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:53:05.768253 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:05.768225 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:53:06.150615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:06.150558 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 29 22:53:06.150901 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:06.150869 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 29 22:53:07.146603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:07.145938 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 29 22:53:07.146603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:07.146002 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 29 22:53:07.879634 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:07.879573 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:53:07.892291 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:07.892268 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:53:08.148056 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:08.147962 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 29 22:53:08.148686 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:08.148647 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 29 22:53:08.785233 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:08.785200 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:53:09.145056 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:09.144958 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 29 22:53:09.145316 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:09.145282 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 29 22:53:10.149217 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.149190 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:53:10.150112 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.150095 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 22:53:10.346285 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.346242 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw"] Oct 29 22:53:10.348884 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.348858 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.362782 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.362755 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"openshift-service-ca.crt\"" Oct 29 22:53:10.362908 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.362821 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"kube-root-ca.crt\"" Oct 29 22:53:10.362908 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.362876 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"default-dockercfg-v8lk4\"" Oct 29 22:53:10.362987 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.362948 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"simulator-w5-2077b4-kserve-self-signed-certs\"" Oct 29 22:53:10.372898 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.372854 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw"] Oct 29 22:53:10.408460 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.408385 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-home\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.408460 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.408452 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2rf\" (UniqueName: \"kubernetes.io/projected/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-kube-api-access-nq2rf\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.408688 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.408525 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tls-certs\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.408688 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.408560 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tmp-dir\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.408688 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.408655 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-dshm\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.408688 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.408688 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-model-cache\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.510021 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.509988 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-home\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.510211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.510072 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2rf\" (UniqueName: \"kubernetes.io/projected/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-kube-api-access-nq2rf\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.510211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.510105 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tls-certs\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.510211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.510133 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tmp-dir\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.510211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.510166 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-dshm\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.510211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.510199 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-model-cache\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.510492 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.510434 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-home\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.510535 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.510507 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tmp-dir\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.510617 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.510596 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-model-cache\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.512538 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.512513 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-dshm\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.512920 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.512897 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tls-certs\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.532763 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.532739 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2rf\" (UniqueName: \"kubernetes.io/projected/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-kube-api-access-nq2rf\") pod \"simulator-w5-2077b4-kserve-79546bd4db-pxmkw\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.659989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.659888 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:10.956395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:10.956308 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw"] Oct 29 22:53:10.960770 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:53:10.960743 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe63a674_17d5_4a5a_ac1c_e76de31fd12d.slice/crio-c6b86ebb5ad372b8a948ccfd804ff064e784be799a770b7b21efe1cc47184efd WatchSource:0}: Error finding container c6b86ebb5ad372b8a948ccfd804ff064e784be799a770b7b21efe1cc47184efd: Status 404 returned error can't find the container with id c6b86ebb5ad372b8a948ccfd804ff064e784be799a770b7b21efe1cc47184efd Oct 29 22:53:11.799162 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:11.799124 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" event={"ID":"fe63a674-17d5-4a5a-ac1c-e76de31fd12d","Type":"ContainerStarted","Data":"70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38"} Oct 29 22:53:11.799649 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:11.799175 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" event={"ID":"fe63a674-17d5-4a5a-ac1c-e76de31fd12d","Type":"ContainerStarted","Data":"c6b86ebb5ad372b8a948ccfd804ff064e784be799a770b7b21efe1cc47184efd"} Oct 29 22:53:11.837637 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:11.837575 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podStartSLOduration=1.837562259 podStartE2EDuration="1.837562259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:11.837355207 +0000 UTC m=+588.953274817" watchObservedRunningTime="2026-10-29 22:53:11.837562259 +0000 UTC m=+588.953481858" Oct 29 22:53:12.660773 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:12.660728 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:16.257725 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.257677 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz"] Oct 29 22:53:16.260442 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.260422 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.284391 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.284364 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"openshift-service-ca.crt\"" Oct 29 22:53:16.284572 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.284554 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"kube-root-ca.crt\"" Oct 29 22:53:16.284677 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.284656 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"default-dockercfg-2lnwl\"" Oct 29 22:53:16.284915 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.284892 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"simulator-w6-1ee571-kserve-self-signed-certs\"" Oct 29 22:53:16.286916 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.286890 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz"] Oct 29 22:53:16.373486 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.373444 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-model-cache\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.373687 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.373498 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-dshm\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.373687 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.373567 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-home\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.373687 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.373619 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tmp-dir\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.373871 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.373723 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tls-certs\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.373871 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.373750 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflt5\" (UniqueName: \"kubernetes.io/projected/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-kube-api-access-wflt5\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.474661 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.474616 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-model-cache\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.474857 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.474689 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-dshm\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.474857 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.474712 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-home\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.474857 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.474729 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tmp-dir\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.474857 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.474809 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tls-certs\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.474857 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.474849 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflt5\" (UniqueName: \"kubernetes.io/projected/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-kube-api-access-wflt5\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.475131 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.475059 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-model-cache\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.475491 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.475466 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tmp-dir\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.475710 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.475467 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-home\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.477450 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.477422 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-dshm\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.477620 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.477502 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tls-certs\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.496502 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.496473 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflt5\" (UniqueName: \"kubernetes.io/projected/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-kube-api-access-wflt5\") pod \"simulator-w6-1ee571-kserve-5db5446bcc-xh9sz\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.574379 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.574285 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:16.875168 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:16.875095 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz"] Oct 29 22:53:16.879080 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:53:16.879050 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87b0f10e_ea68_4a85_a6d9_01a277a4b5b4.slice/crio-295cb1fed0ecf1901f8781916bd3823eaaef59c9499dee6c50b2bd9be73446cd WatchSource:0}: Error finding container 295cb1fed0ecf1901f8781916bd3823eaaef59c9499dee6c50b2bd9be73446cd: Status 404 returned error can't find the container with id 295cb1fed0ecf1901f8781916bd3823eaaef59c9499dee6c50b2bd9be73446cd Oct 29 22:53:17.837569 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:17.837531 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" event={"ID":"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4","Type":"ContainerStarted","Data":"fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc"} Oct 29 22:53:17.842696 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:17.837575 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" event={"ID":"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4","Type":"ContainerStarted","Data":"295cb1fed0ecf1901f8781916bd3823eaaef59c9499dee6c50b2bd9be73446cd"} Oct 29 22:53:17.878797 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:17.878748 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podStartSLOduration=1.8787333510000002 podStartE2EDuration="1.878733351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:17.876700938 +0000 UTC m=+594.992620543" watchObservedRunningTime="2026-10-29 22:53:17.878733351 +0000 UTC m=+594.994652957" Oct 29 22:53:18.574750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:18.574718 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:20.660430 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:20.660393 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:20.679323 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:20.679300 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:20.868356 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:20.868330 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:53:23.064596 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:23.064556 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:53:23.065128 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:23.065107 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:53:26.054607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.054538 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56"] Oct 29 22:53:26.057941 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.057922 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.074737 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.074718 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"openshift-service-ca.crt\"" Oct 29 22:53:26.085417 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.085389 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"default-dockercfg-c28wt\"" Oct 29 22:53:26.085531 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.085485 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"simulator-w2-7aeaad-kserve-self-signed-certs\"" Oct 29 22:53:26.086929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.086904 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56"] Oct 29 22:53:26.175428 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.175397 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-model-cache\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.175569 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.175438 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d017043-51f9-4ade-9f6b-fa48f7066093-tls-certs\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.175645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.175625 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-dshm\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.175706 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.175687 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-tmp-dir\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.175801 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.175785 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-home\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.175843 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.175816 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnzm7\" (UniqueName: \"kubernetes.io/projected/7d017043-51f9-4ade-9f6b-fa48f7066093-kube-api-access-nnzm7\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.179205 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.179186 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"kube-root-ca.crt\"" Oct 29 22:53:26.277216 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.277180 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-dshm\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.277419 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.277235 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-tmp-dir\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.277419 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.277293 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-home\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.277763 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.277731 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzm7\" (UniqueName: \"kubernetes.io/projected/7d017043-51f9-4ade-9f6b-fa48f7066093-kube-api-access-nnzm7\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.277763 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.277755 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-tmp-dir\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.277960 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.277766 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-home\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.277960 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.277866 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-model-cache\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.277960 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.277902 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d017043-51f9-4ade-9f6b-fa48f7066093-tls-certs\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.278274 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.278254 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-model-cache\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.281001 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.280972 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d017043-51f9-4ade-9f6b-fa48f7066093-tls-certs\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.297884 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.297850 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-dshm\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.299395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.299372 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnzm7\" (UniqueName: \"kubernetes.io/projected/7d017043-51f9-4ade-9f6b-fa48f7066093-kube-api-access-nnzm7\") pod \"simulator-w2-7aeaad-kserve-64d79ddd64-gpv56\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.370627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.370519 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:26.574709 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.574675 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:26.587434 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.587412 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:26.673738 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.673654 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56"] Oct 29 22:53:26.677057 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:53:26.677027 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d017043_51f9_4ade_9f6b_fa48f7066093.slice/crio-600979e338400fe9a19f09b84bd7e9a451e23af473d8a0b981ae3a7c398ba833 WatchSource:0}: Error finding container 600979e338400fe9a19f09b84bd7e9a451e23af473d8a0b981ae3a7c398ba833: Status 404 returned error can't find the container with id 600979e338400fe9a19f09b84bd7e9a451e23af473d8a0b981ae3a7c398ba833 Oct 29 22:53:26.822298 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:53:26.822266 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:53:26.890989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.890958 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" event={"ID":"7d017043-51f9-4ade-9f6b-fa48f7066093","Type":"ContainerStarted","Data":"78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d"} Oct 29 22:53:26.891147 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.891001 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" event={"ID":"7d017043-51f9-4ade-9f6b-fa48f7066093","Type":"ContainerStarted","Data":"600979e338400fe9a19f09b84bd7e9a451e23af473d8a0b981ae3a7c398ba833"} Oct 29 22:53:26.902420 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.902396 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:53:26.930028 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:26.929945 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podStartSLOduration=0.929927428 podStartE2EDuration="929.927428ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:26.928739732 +0000 UTC m=+604.044659337" watchObservedRunningTime="2026-10-29 22:53:26.929927428 +0000 UTC m=+604.045847036" Oct 29 22:53:27.370916 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:27.370879 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:30.899746 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:30.899709 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw"] Oct 29 22:53:30.902425 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:30.902404 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:30.918119 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:30.918090 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"default-dockercfg-bdgrh\"" Oct 29 22:53:30.928682 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:30.928656 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"simulator-w4-ed8218-kserve-self-signed-certs\"" Oct 29 22:53:30.930594 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:30.930558 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw"] Oct 29 22:53:30.934103 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:30.934071 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"kube-root-ca.crt\"" Oct 29 22:53:30.934228 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:30.934111 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"openshift-service-ca.crt\"" Oct 29 22:53:31.028551 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.028514 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-home\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.028732 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.028555 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqd9\" (UniqueName: \"kubernetes.io/projected/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-kube-api-access-vqqd9\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.028732 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.028686 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tmp-dir\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.028732 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.028725 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-model-cache\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.028913 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.028887 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-dshm\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.029018 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.029003 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tls-certs\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.133627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.131234 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tls-certs\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.133627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.131293 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-home\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.133627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.131327 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqd9\" (UniqueName: \"kubernetes.io/projected/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-kube-api-access-vqqd9\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.133627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.131399 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tmp-dir\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.133627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.131450 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-model-cache\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.133627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.131531 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-dshm\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.133627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.131856 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tmp-dir\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.133627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.132125 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-home\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.133627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.132165 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-model-cache\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.143127 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.143090 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tls-certs\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.150903 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.150836 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-dshm\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.153273 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.153248 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqd9\" (UniqueName: \"kubernetes.io/projected/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-kube-api-access-vqqd9\") pod \"simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.215309 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.215276 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:31.520672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.520636 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw"] Oct 29 22:53:31.523963 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:53:31.523934 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda35990d6_4dc3_477c_83cd_14fcf9a3a84b.slice/crio-2eeb9f3199cbad31975e60d5e1e71248d6e129e0857da432194fef2ae2093db7 WatchSource:0}: Error finding container 2eeb9f3199cbad31975e60d5e1e71248d6e129e0857da432194fef2ae2093db7: Status 404 returned error can't find the container with id 2eeb9f3199cbad31975e60d5e1e71248d6e129e0857da432194fef2ae2093db7 Oct 29 22:53:31.919667 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.919568 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" event={"ID":"a35990d6-4dc3-477c-83cd-14fcf9a3a84b","Type":"ContainerStarted","Data":"6c2774cb33fb02be2d161612a0e7380f12301d00bbd53155c432006c958bd171"} Oct 29 22:53:31.919667 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.919634 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" event={"ID":"a35990d6-4dc3-477c-83cd-14fcf9a3a84b","Type":"ContainerStarted","Data":"2eeb9f3199cbad31975e60d5e1e71248d6e129e0857da432194fef2ae2093db7"} Oct 29 22:53:31.958419 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:31.958294 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podStartSLOduration=1.9582782060000001 podStartE2EDuration="1.958278206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:31.957550897 +0000 UTC m=+609.073470504" watchObservedRunningTime="2026-10-29 22:53:31.958278206 +0000 UTC m=+609.074197813" Oct 29 22:53:32.215799 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:32.215758 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:36.257538 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.257498 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r"] Oct 29 22:53:36.260195 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.260174 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.273341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.273319 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:53:36.283616 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.283569 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:53:36.285976 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.285954 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r"] Oct 29 22:53:36.370940 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.370912 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:36.383155 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.383135 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:36.385659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.385635 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-031adb66-qvm5r\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.385762 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.385748 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-031adb66-qvm5r\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.385837 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.385824 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4v5x\" (UniqueName: \"kubernetes.io/projected/b30c8d44-74b9-4582-b453-b6b214e6ea93-kube-api-access-v4v5x\") pod \"maas-api-revoke-sub-031adb66-qvm5r\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.487137 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.487100 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-031adb66-qvm5r\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.487296 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.487190 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4v5x\" (UniqueName: \"kubernetes.io/projected/b30c8d44-74b9-4582-b453-b6b214e6ea93-kube-api-access-v4v5x\") pod \"maas-api-revoke-sub-031adb66-qvm5r\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.487296 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.487260 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-031adb66-qvm5r\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.487914 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.487890 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-031adb66-qvm5r\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.489545 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.489524 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-031adb66-qvm5r\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.510964 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.510909 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4v5x\" (UniqueName: \"kubernetes.io/projected/b30c8d44-74b9-4582-b453-b6b214e6ea93-kube-api-access-v4v5x\") pod \"maas-api-revoke-sub-031adb66-qvm5r\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.572887 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.572861 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:36.871828 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.871739 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r"] Oct 29 22:53:36.875799 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:53:36.875771 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30c8d44_74b9_4582_b453_b6b214e6ea93.slice/crio-cd65b38ed0a65db6ee32f10a97fda653f3be5d607f1fe66656b7a664c9fa375c WatchSource:0}: Error finding container cd65b38ed0a65db6ee32f10a97fda653f3be5d607f1fe66656b7a664c9fa375c: Status 404 returned error can't find the container with id cd65b38ed0a65db6ee32f10a97fda653f3be5d607f1fe66656b7a664c9fa375c Oct 29 22:53:36.946638 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.946610 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" event={"ID":"b30c8d44-74b9-4582-b453-b6b214e6ea93","Type":"ContainerStarted","Data":"cd65b38ed0a65db6ee32f10a97fda653f3be5d607f1fe66656b7a664c9fa375c"} Oct 29 22:53:36.957799 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:36.957778 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:53:38.763538 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.763496 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c"] Oct 29 22:53:38.771858 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.771830 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:38.789448 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.789413 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c"] Oct 29 22:53:38.798261 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.798234 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"premium-w5-2077b4-kserve-self-signed-certs\"" Oct 29 22:53:38.913030 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.912999 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-dshm\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:38.913167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.913088 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-home\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:38.913167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.913160 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-model-cache\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:38.913253 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.913186 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tls-certs\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:38.913253 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.913225 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nkgq\" (UniqueName: \"kubernetes.io/projected/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-kube-api-access-4nkgq\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:38.913415 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:38.913396 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tmp-dir\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.015294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.015185 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tmp-dir\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.015460 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.015355 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-dshm\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.015460 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.015390 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-home\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.015460 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.015441 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-model-cache\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.015658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.015467 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tls-certs\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.015658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.015507 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nkgq\" (UniqueName: \"kubernetes.io/projected/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-kube-api-access-4nkgq\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.015872 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.015844 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tmp-dir\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.015968 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.015895 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-home\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.016264 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.016232 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-model-cache\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.018169 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.018146 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-dshm\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.019138 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.019115 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tls-certs\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.037604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.037551 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nkgq\" (UniqueName: \"kubernetes.io/projected/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-kube-api-access-4nkgq\") pod \"premium-w5-2077b4-kserve-796cdd8f59-xct2c\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.087436 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.087406 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:39.386923 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.386846 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c"] Oct 29 22:53:39.390808 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:53:39.390772 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc19011b8_e3f1_4fb5_81a8_fa98c5f0a156.slice/crio-9164c22b800570e88dfd796dadaf139a1bcb24cbe9ec8cb050c0f93cb3d8e0ea WatchSource:0}: Error finding container 9164c22b800570e88dfd796dadaf139a1bcb24cbe9ec8cb050c0f93cb3d8e0ea: Status 404 returned error can't find the container with id 9164c22b800570e88dfd796dadaf139a1bcb24cbe9ec8cb050c0f93cb3d8e0ea Oct 29 22:53:39.969449 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.969415 5736 generic.go:356] "Generic (PLEG): container finished" podID="b30c8d44-74b9-4582-b453-b6b214e6ea93" containerID="fe412e0ecad3632b6379e2743f022e9ea60d7bcfa53b1b4e58d72304645cdcd5" exitCode=0 Oct 29 22:53:39.969955 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.969516 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" event={"ID":"b30c8d44-74b9-4582-b453-b6b214e6ea93","Type":"ContainerDied","Data":"fe412e0ecad3632b6379e2743f022e9ea60d7bcfa53b1b4e58d72304645cdcd5"} Oct 29 22:53:39.971624 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.971569 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" event={"ID":"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156","Type":"ContainerStarted","Data":"0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2"} Oct 29 22:53:39.971746 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:39.971638 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" event={"ID":"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156","Type":"ContainerStarted","Data":"9164c22b800570e88dfd796dadaf139a1bcb24cbe9ec8cb050c0f93cb3d8e0ea"} Oct 29 22:53:40.045372 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:40.045269 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podStartSLOduration=2.045254049 podStartE2EDuration="2.045254049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:40.044039195 +0000 UTC m=+617.159958806" watchObservedRunningTime="2026-10-29 22:53:40.045254049 +0000 UTC m=+617.161173699" Oct 29 22:53:40.087972 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:40.087926 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:41.124695 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.124672 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:41.135951 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.135931 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30c8d44-74b9-4582-b453-b6b214e6ea93-kube-api-access-v4v5x\" (UniqueName: \"kubernetes.io/projected/b30c8d44-74b9-4582-b453-b6b214e6ea93-kube-api-access-v4v5x\") pod \"b30c8d44-74b9-4582-b453-b6b214e6ea93\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " Oct 29 22:53:41.136016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.135969 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-xks-ca\") pod \"b30c8d44-74b9-4582-b453-b6b214e6ea93\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " Oct 29 22:53:41.136053 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.136012 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-openshift-ca\") pod \"b30c8d44-74b9-4582-b453-b6b214e6ea93\" (UID: \"b30c8d44-74b9-4582-b453-b6b214e6ea93\") " Oct 29 22:53:41.136386 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.136357 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-openshift-ca" pod "b30c8d44-74b9-4582-b453-b6b214e6ea93" (UID: "b30c8d44-74b9-4582-b453-b6b214e6ea93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:41.136525 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.136511 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:53:41.138062 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.138040 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-xks-ca" pod "b30c8d44-74b9-4582-b453-b6b214e6ea93" (UID: "b30c8d44-74b9-4582-b453-b6b214e6ea93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:41.138148 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.138120 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30c8d44-74b9-4582-b453-b6b214e6ea93-kube-api-access-v4v5x" pod "b30c8d44-74b9-4582-b453-b6b214e6ea93" (UID: "b30c8d44-74b9-4582-b453-b6b214e6ea93"). InnerVolumeSpecName "kube-api-access-v4v5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:41.215698 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.215663 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:41.227983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.227958 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:41.237733 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.237669 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4v5x\" (UniqueName: \"kubernetes.io/projected/b30c8d44-74b9-4582-b453-b6b214e6ea93-kube-api-access-v4v5x\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:53:41.237733 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.237692 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30c8d44-74b9-4582-b453-b6b214e6ea93-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:53:41.984957 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.984917 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" Oct 29 22:53:41.985142 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.984947 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r" event={"ID":"b30c8d44-74b9-4582-b453-b6b214e6ea93","Type":"ContainerDied","Data":"cd65b38ed0a65db6ee32f10a97fda653f3be5d607f1fe66656b7a664c9fa375c"} Oct 29 22:53:41.985142 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.985025 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd65b38ed0a65db6ee32f10a97fda653f3be5d607f1fe66656b7a664c9fa375c" Oct 29 22:53:41.996518 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:41.996495 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:53:49.088099 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.088068 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:49.101495 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.101476 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:49.818658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.818623 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm"] Oct 29 22:53:49.819132 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.819111 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b30c8d44-74b9-4582-b453-b6b214e6ea93" containerName="revoke-keys" Oct 29 22:53:49.819193 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.819133 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30c8d44-74b9-4582-b453-b6b214e6ea93" containerName="revoke-keys" Oct 29 22:53:49.819297 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.819285 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30c8d44-74b9-4582-b453-b6b214e6ea93" containerName="revoke-keys" Oct 29 22:53:49.824666 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.824646 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:49.850215 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.850192 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"premium-w6-1ee571-kserve-self-signed-certs\"" Oct 29 22:53:49.852896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.852872 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm"] Oct 29 22:53:49.926241 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.926208 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-home\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:49.926407 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.926256 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-tmp-dir\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:49.926407 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.926367 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-dshm\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:49.926487 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.926420 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1aa2b1-192a-485e-9791-a71051fe0172-tls-certs\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:49.926554 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.926540 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-model-cache\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:49.926621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:49.926575 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7txz\" (UniqueName: \"kubernetes.io/projected/2a1aa2b1-192a-485e-9791-a71051fe0172-kube-api-access-x7txz\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.027465 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.027423 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-model-cache\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.027730 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.027497 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7txz\" (UniqueName: \"kubernetes.io/projected/2a1aa2b1-192a-485e-9791-a71051fe0172-kube-api-access-x7txz\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.027730 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.027562 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-home\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.027730 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.027649 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-tmp-dir\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.027730 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.027705 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-dshm\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.027963 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.027747 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1aa2b1-192a-485e-9791-a71051fe0172-tls-certs\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.027963 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.027857 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-model-cache\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.028124 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.028098 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-home\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.028426 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.028400 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-tmp-dir\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.031020 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.030995 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-dshm\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.031144 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.031019 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1aa2b1-192a-485e-9791-a71051fe0172-tls-certs\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.042673 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.042647 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:53:50.050097 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.050065 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7txz\" (UniqueName: \"kubernetes.io/projected/2a1aa2b1-192a-485e-9791-a71051fe0172-kube-api-access-x7txz\") pod \"premium-w6-1ee571-kserve-846f9b65f6-cxpdm\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.143404 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.143333 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:50.443954 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.443872 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm"] Oct 29 22:53:50.447541 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:53:50.447514 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a1aa2b1_192a_485e_9791_a71051fe0172.slice/crio-881e81d7934ba11b71cdeb2b983ba9451d9513d0c64f2e01838902c10bac7695 WatchSource:0}: Error finding container 881e81d7934ba11b71cdeb2b983ba9451d9513d0c64f2e01838902c10bac7695: Status 404 returned error can't find the container with id 881e81d7934ba11b71cdeb2b983ba9451d9513d0c64f2e01838902c10bac7695 Oct 29 22:53:50.713284 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.713252 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm"] Oct 29 22:53:50.716984 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.716966 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:50.730373 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.730352 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:53:50.730486 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.730376 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:53:50.737660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.737630 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm"] Oct 29 22:53:50.835879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.835853 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad74f549-96dsm\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:50.836010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.835906 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad74f549-96dsm\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:50.836010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.835984 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwtj\" (UniqueName: \"kubernetes.io/projected/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-kube-api-access-thwtj\") pod \"maas-api-revoke-sub-ad74f549-96dsm\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:50.937252 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.937228 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad74f549-96dsm\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:50.937389 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.937262 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwtj\" (UniqueName: \"kubernetes.io/projected/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-kube-api-access-thwtj\") pod \"maas-api-revoke-sub-ad74f549-96dsm\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:50.937389 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.937361 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad74f549-96dsm\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:50.937977 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.937956 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad74f549-96dsm\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:50.939589 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.939560 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad74f549-96dsm\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:50.959330 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:50.959306 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwtj\" (UniqueName: \"kubernetes.io/projected/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-kube-api-access-thwtj\") pod \"maas-api-revoke-sub-ad74f549-96dsm\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:51.029996 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:51.029919 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:51.036715 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:51.036680 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" event={"ID":"2a1aa2b1-192a-485e-9791-a71051fe0172","Type":"ContainerStarted","Data":"46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e"} Oct 29 22:53:51.036823 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:51.036734 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" event={"ID":"2a1aa2b1-192a-485e-9791-a71051fe0172","Type":"ContainerStarted","Data":"881e81d7934ba11b71cdeb2b983ba9451d9513d0c64f2e01838902c10bac7695"} Oct 29 22:53:51.078601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:51.077498 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podStartSLOduration=2.077475611 podStartE2EDuration="2.077475611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:51.075266985 +0000 UTC m=+628.191186591" watchObservedRunningTime="2026-10-29 22:53:51.077475611 +0000 UTC m=+628.193395218" Oct 29 22:53:51.143982 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:51.143957 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:53:51.336743 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:51.336669 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm"] Oct 29 22:53:51.341044 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:53:51.341014 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591a40e8_e29f_4fcf_a3e5_945ffd2efc5a.slice/crio-8c07a46ca59a0f8c69f594bb4c4af8305c8b5f7975b40cbb54e02c744e6c79f2 WatchSource:0}: Error finding container 8c07a46ca59a0f8c69f594bb4c4af8305c8b5f7975b40cbb54e02c744e6c79f2: Status 404 returned error can't find the container with id 8c07a46ca59a0f8c69f594bb4c4af8305c8b5f7975b40cbb54e02c744e6c79f2 Oct 29 22:53:52.045146 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:52.045112 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" event={"ID":"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a","Type":"ContainerStarted","Data":"40c75c0f2af3e233fc4183a6527bc42e75d0cb3baa12c6569275cd55ee0147eb"} Oct 29 22:53:52.045319 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:52.045175 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" event={"ID":"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a","Type":"ContainerStarted","Data":"8c07a46ca59a0f8c69f594bb4c4af8305c8b5f7975b40cbb54e02c744e6c79f2"} Oct 29 22:53:52.083069 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:52.083020 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" podStartSLOduration=2.083007561 podStartE2EDuration="2.083007561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:52.082692205 +0000 UTC m=+629.198611813" watchObservedRunningTime="2026-10-29 22:53:52.083007561 +0000 UTC m=+629.198927209" Oct 29 22:53:53.052318 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:53.052283 5736 generic.go:356] "Generic (PLEG): container finished" podID="591a40e8-e29f-4fcf-a3e5-945ffd2efc5a" containerID="40c75c0f2af3e233fc4183a6527bc42e75d0cb3baa12c6569275cd55ee0147eb" exitCode=0 Oct 29 22:53:53.052751 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:53.052363 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" event={"ID":"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a","Type":"ContainerDied","Data":"40c75c0f2af3e233fc4183a6527bc42e75d0cb3baa12c6569275cd55ee0147eb"} Oct 29 22:53:54.202534 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.202508 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:54.273365 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.273331 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-openshift-ca\") pod \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " Oct 29 22:53:54.273530 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.273377 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-xks-ca\") pod \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " Oct 29 22:53:54.273530 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.273425 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-kube-api-access-thwtj\" (UniqueName: \"kubernetes.io/projected/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-kube-api-access-thwtj\") pod \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\" (UID: \"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a\") " Oct 29 22:53:54.273863 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.273794 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-openshift-ca" pod "591a40e8-e29f-4fcf-a3e5-945ffd2efc5a" (UID: "591a40e8-e29f-4fcf-a3e5-945ffd2efc5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:54.275654 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.275621 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-xks-ca" pod "591a40e8-e29f-4fcf-a3e5-945ffd2efc5a" (UID: "591a40e8-e29f-4fcf-a3e5-945ffd2efc5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:54.275746 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.275707 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-kube-api-access-thwtj" pod "591a40e8-e29f-4fcf-a3e5-945ffd2efc5a" (UID: "591a40e8-e29f-4fcf-a3e5-945ffd2efc5a"). InnerVolumeSpecName "kube-api-access-thwtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:54.374866 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.374804 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:53:54.374866 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.374834 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:53:54.374866 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:54.374844 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thwtj\" (UniqueName: \"kubernetes.io/projected/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a-kube-api-access-thwtj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:53:55.065074 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:55.065048 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" Oct 29 22:53:55.065236 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:55.065042 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm" event={"ID":"591a40e8-e29f-4fcf-a3e5-945ffd2efc5a","Type":"ContainerDied","Data":"8c07a46ca59a0f8c69f594bb4c4af8305c8b5f7975b40cbb54e02c744e6c79f2"} Oct 29 22:53:55.065236 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:55.065188 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c07a46ca59a0f8c69f594bb4c4af8305c8b5f7975b40cbb54e02c744e6c79f2" Oct 29 22:53:56.826333 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:53:56.826294 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:53:59.348848 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.348796 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl"] Oct 29 22:53:59.349367 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.349349 5736 cpu_manager.go:414] "containerMap: removing container" podUID="591a40e8-e29f-4fcf-a3e5-945ffd2efc5a" containerName="revoke-keys" Oct 29 22:53:59.349417 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.349369 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591a40e8-e29f-4fcf-a3e5-945ffd2efc5a" containerName="revoke-keys" Oct 29 22:53:59.349456 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.349452 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="591a40e8-e29f-4fcf-a3e5-945ffd2efc5a" containerName="revoke-keys" Oct 29 22:53:59.354976 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.354953 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.370810 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.370790 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:53:59.372709 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.372687 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl"] Oct 29 22:53:59.381480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.381459 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:53:59.440122 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.440057 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a88eb12-s8dtl\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.440225 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.440204 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a88eb12-s8dtl\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.440311 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.440296 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n999\" (UniqueName: \"kubernetes.io/projected/a70e218d-0c77-495c-8235-1eb634ea9739-kube-api-access-2n999\") pod \"maas-api-revoke-sub-0a88eb12-s8dtl\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.541862 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.541836 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n999\" (UniqueName: \"kubernetes.io/projected/a70e218d-0c77-495c-8235-1eb634ea9739-kube-api-access-2n999\") pod \"maas-api-revoke-sub-0a88eb12-s8dtl\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.541983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.541887 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a88eb12-s8dtl\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.541983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.541946 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a88eb12-s8dtl\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.542527 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.542503 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a88eb12-s8dtl\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.544167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.544147 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a88eb12-s8dtl\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.564105 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.564081 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n999\" (UniqueName: \"kubernetes.io/projected/a70e218d-0c77-495c-8235-1eb634ea9739-kube-api-access-2n999\") pod \"maas-api-revoke-sub-0a88eb12-s8dtl\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.667369 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.667343 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:53:59.969977 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:53:59.969905 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl"] Oct 29 22:53:59.973778 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:53:59.973747 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70e218d_0c77_495c_8235_1eb634ea9739.slice/crio-b37a15bf6fb71ef4a084a192ea5c1521d644fdc7c66394d0739d3db870758fa7 WatchSource:0}: Error finding container b37a15bf6fb71ef4a084a192ea5c1521d644fdc7c66394d0739d3db870758fa7: Status 404 returned error can't find the container with id b37a15bf6fb71ef4a084a192ea5c1521d644fdc7c66394d0739d3db870758fa7 Oct 29 22:54:00.093153 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:00.093110 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" event={"ID":"a70e218d-0c77-495c-8235-1eb634ea9739","Type":"ContainerStarted","Data":"b37a15bf6fb71ef4a084a192ea5c1521d644fdc7c66394d0739d3db870758fa7"} Oct 29 22:54:00.143736 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:00.143707 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:54:00.156761 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:00.156737 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:54:01.100083 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:01.100004 5736 generic.go:356] "Generic (PLEG): container finished" podID="a70e218d-0c77-495c-8235-1eb634ea9739" containerID="9158d28675aac49e173824d0760d36b7c450864d134b9e6d23a279bb58eae063" exitCode=0 Oct 29 22:54:01.100497 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:01.100092 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" event={"ID":"a70e218d-0c77-495c-8235-1eb634ea9739","Type":"ContainerDied","Data":"9158d28675aac49e173824d0760d36b7c450864d134b9e6d23a279bb58eae063"} Oct 29 22:54:01.111440 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:01.111416 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:54:02.248080 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.248050 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:54:02.372824 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.372745 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-openshift-ca\") pod \"a70e218d-0c77-495c-8235-1eb634ea9739\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " Oct 29 22:54:02.372963 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.372864 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70e218d-0c77-495c-8235-1eb634ea9739-kube-api-access-2n999\" (UniqueName: \"kubernetes.io/projected/a70e218d-0c77-495c-8235-1eb634ea9739-kube-api-access-2n999\") pod \"a70e218d-0c77-495c-8235-1eb634ea9739\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " Oct 29 22:54:02.372963 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.372889 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-xks-ca\") pod \"a70e218d-0c77-495c-8235-1eb634ea9739\" (UID: \"a70e218d-0c77-495c-8235-1eb634ea9739\") " Oct 29 22:54:02.373155 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.373130 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-openshift-ca" pod "a70e218d-0c77-495c-8235-1eb634ea9739" (UID: "a70e218d-0c77-495c-8235-1eb634ea9739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:02.373302 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.373281 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:02.374902 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.374878 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-xks-ca" pod "a70e218d-0c77-495c-8235-1eb634ea9739" (UID: "a70e218d-0c77-495c-8235-1eb634ea9739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:02.374985 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.374952 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70e218d-0c77-495c-8235-1eb634ea9739-kube-api-access-2n999" pod "a70e218d-0c77-495c-8235-1eb634ea9739" (UID: "a70e218d-0c77-495c-8235-1eb634ea9739"). InnerVolumeSpecName "kube-api-access-2n999". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:02.474111 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.474078 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n999\" (UniqueName: \"kubernetes.io/projected/a70e218d-0c77-495c-8235-1eb634ea9739-kube-api-access-2n999\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:02.474111 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:02.474109 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70e218d-0c77-495c-8235-1eb634ea9739-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:03.014621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.014561 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt"] Oct 29 22:54:03.015216 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.015191 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a70e218d-0c77-495c-8235-1eb634ea9739" containerName="revoke-keys" Oct 29 22:54:03.015368 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.015351 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70e218d-0c77-495c-8235-1eb634ea9739" containerName="revoke-keys" Oct 29 22:54:03.015610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.015575 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70e218d-0c77-495c-8235-1eb634ea9739" containerName="revoke-keys" Oct 29 22:54:03.018894 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.018876 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.041916 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.041880 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt"] Oct 29 22:54:03.121896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.121867 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" Oct 29 22:54:03.122057 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.121898 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl" event={"ID":"a70e218d-0c77-495c-8235-1eb634ea9739","Type":"ContainerDied","Data":"b37a15bf6fb71ef4a084a192ea5c1521d644fdc7c66394d0739d3db870758fa7"} Oct 29 22:54:03.122057 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.121946 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37a15bf6fb71ef4a084a192ea5c1521d644fdc7c66394d0739d3db870758fa7" Oct 29 22:54:03.182624 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.182572 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c90888-m7jvt\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.182842 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.182729 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdfg\" (UniqueName: \"kubernetes.io/projected/acd25c55-e25f-41a2-be9c-96ee70cf06da-kube-api-access-mzdfg\") pod \"maas-api-revoke-sub-20c90888-m7jvt\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.182889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.182857 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c90888-m7jvt\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.283811 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.283728 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c90888-m7jvt\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.283811 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.283792 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c90888-m7jvt\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.284189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.283855 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdfg\" (UniqueName: \"kubernetes.io/projected/acd25c55-e25f-41a2-be9c-96ee70cf06da-kube-api-access-mzdfg\") pod \"maas-api-revoke-sub-20c90888-m7jvt\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.284524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.284505 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c90888-m7jvt\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.286016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.285996 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c90888-m7jvt\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.305532 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.305507 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdfg\" (UniqueName: \"kubernetes.io/projected/acd25c55-e25f-41a2-be9c-96ee70cf06da-kube-api-access-mzdfg\") pod \"maas-api-revoke-sub-20c90888-m7jvt\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.332375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.332348 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:03.632832 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:03.632746 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt"] Oct 29 22:54:03.636617 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:54:03.636575 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd25c55_e25f_41a2_be9c_96ee70cf06da.slice/crio-416b68e0aaaf1d9e17a97d23800489dc64e04e601e5636d872bd7cd5074ab795 WatchSource:0}: Error finding container 416b68e0aaaf1d9e17a97d23800489dc64e04e601e5636d872bd7cd5074ab795: Status 404 returned error can't find the container with id 416b68e0aaaf1d9e17a97d23800489dc64e04e601e5636d872bd7cd5074ab795 Oct 29 22:54:04.128943 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:04.128908 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" event={"ID":"acd25c55-e25f-41a2-be9c-96ee70cf06da","Type":"ContainerStarted","Data":"31de1efa31ce48ada8b46234ae429b44c9d51efaf0185ff2b0e5009cae29bb57"} Oct 29 22:54:04.129148 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:04.128964 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" event={"ID":"acd25c55-e25f-41a2-be9c-96ee70cf06da","Type":"ContainerStarted","Data":"416b68e0aaaf1d9e17a97d23800489dc64e04e601e5636d872bd7cd5074ab795"} Oct 29 22:54:04.178318 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:04.178266 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" podStartSLOduration=2.178250825 podStartE2EDuration="2.178250825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:04.164910342 +0000 UTC m=+641.280829948" watchObservedRunningTime="2026-10-29 22:54:04.178250825 +0000 UTC m=+641.294170430" Oct 29 22:54:05.135558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:05.135525 5736 generic.go:356] "Generic (PLEG): container finished" podID="acd25c55-e25f-41a2-be9c-96ee70cf06da" containerID="31de1efa31ce48ada8b46234ae429b44c9d51efaf0185ff2b0e5009cae29bb57" exitCode=0 Oct 29 22:54:05.135967 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:05.135611 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" event={"ID":"acd25c55-e25f-41a2-be9c-96ee70cf06da","Type":"ContainerDied","Data":"31de1efa31ce48ada8b46234ae429b44c9d51efaf0185ff2b0e5009cae29bb57"} Oct 29 22:54:06.289288 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.289263 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:06.420917 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.420833 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-xks-ca\") pod \"acd25c55-e25f-41a2-be9c-96ee70cf06da\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " Oct 29 22:54:06.420917 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.420894 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-openshift-ca\") pod \"acd25c55-e25f-41a2-be9c-96ee70cf06da\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " Oct 29 22:54:06.421146 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.420926 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd25c55-e25f-41a2-be9c-96ee70cf06da-kube-api-access-mzdfg\" (UniqueName: \"kubernetes.io/projected/acd25c55-e25f-41a2-be9c-96ee70cf06da-kube-api-access-mzdfg\") pod \"acd25c55-e25f-41a2-be9c-96ee70cf06da\" (UID: \"acd25c55-e25f-41a2-be9c-96ee70cf06da\") " Oct 29 22:54:06.421277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.421253 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-openshift-ca" pod "acd25c55-e25f-41a2-be9c-96ee70cf06da" (UID: "acd25c55-e25f-41a2-be9c-96ee70cf06da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:06.421376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.421360 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:06.423131 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.423098 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-xks-ca" pod "acd25c55-e25f-41a2-be9c-96ee70cf06da" (UID: "acd25c55-e25f-41a2-be9c-96ee70cf06da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:06.423131 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.423109 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd25c55-e25f-41a2-be9c-96ee70cf06da-kube-api-access-mzdfg" pod "acd25c55-e25f-41a2-be9c-96ee70cf06da" (UID: "acd25c55-e25f-41a2-be9c-96ee70cf06da"). InnerVolumeSpecName "kube-api-access-mzdfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:06.522290 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.522265 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd25c55-e25f-41a2-be9c-96ee70cf06da-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:06.522290 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:06.522290 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzdfg\" (UniqueName: \"kubernetes.io/projected/acd25c55-e25f-41a2-be9c-96ee70cf06da-kube-api-access-mzdfg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:07.147734 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:07.147704 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" Oct 29 22:54:07.147734 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:07.147722 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt" event={"ID":"acd25c55-e25f-41a2-be9c-96ee70cf06da","Type":"ContainerDied","Data":"416b68e0aaaf1d9e17a97d23800489dc64e04e601e5636d872bd7cd5074ab795"} Oct 29 22:54:07.147919 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:07.147767 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="416b68e0aaaf1d9e17a97d23800489dc64e04e601e5636d872bd7cd5074ab795" Oct 29 22:54:14.988428 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:14.988390 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh"] Oct 29 22:54:14.989167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:14.989149 5736 cpu_manager.go:414] "containerMap: removing container" podUID="acd25c55-e25f-41a2-be9c-96ee70cf06da" containerName="revoke-keys" Oct 29 22:54:14.989228 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:14.989171 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd25c55-e25f-41a2-be9c-96ee70cf06da" containerName="revoke-keys" Oct 29 22:54:14.989298 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:14.989286 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd25c55-e25f-41a2-be9c-96ee70cf06da" containerName="revoke-keys" Oct 29 22:54:14.992967 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:14.992950 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.010289 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.010255 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-99dfb0-openshift-default-dockercfg-72qwh\"" Oct 29 22:54:15.013662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.013633 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.013822 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.013805 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.013966 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.013946 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9hzq\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-kube-api-access-g9hzq\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.014100 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.014084 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.014215 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.014201 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.014339 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.014322 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.014536 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.014508 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.014820 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.014796 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.014923 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.014904 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.021069 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.021043 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh"] Oct 29 22:54:15.116501 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.116462 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.116672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.116557 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.116746 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.116675 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.116818 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.116754 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.116818 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.116781 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.116818 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.116807 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9hzq\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-kube-api-access-g9hzq\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.117029 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.116937 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.117103 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.117059 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.117103 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.117064 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.117210 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.117117 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.117210 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.117138 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.117210 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.117164 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.117210 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.117193 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.117627 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.117567 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.119587 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.119551 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.119764 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.119736 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.149883 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.149858 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.150218 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.150194 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9hzq\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-kube-api-access-g9hzq\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.317978 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.317901 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:15.637610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.637517 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh"] Oct 29 22:54:15.641106 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:54:15.641070 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdb0288_46c8_4c6e_a350_beccdd8b6a90.slice/crio-93ae5beba1e8e88ba7e32089d8208c77d84563d0d62a6a227175d2c85e5b0e81 WatchSource:0}: Error finding container 93ae5beba1e8e88ba7e32089d8208c77d84563d0d62a6a227175d2c85e5b0e81: Status 404 returned error can't find the container with id 93ae5beba1e8e88ba7e32089d8208c77d84563d0d62a6a227175d2c85e5b0e81 Oct 29 22:54:15.643614 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.643559 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:15.643699 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.643663 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:15.643735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:15.643707 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:16.197361 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.197326 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" event={"ID":"3fdb0288-46c8-4c6e-a350-beccdd8b6a90","Type":"ContainerStarted","Data":"e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5"} Oct 29 22:54:16.197789 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.197386 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" event={"ID":"3fdb0288-46c8-4c6e-a350-beccdd8b6a90","Type":"ContainerStarted","Data":"93ae5beba1e8e88ba7e32089d8208c77d84563d0d62a6a227175d2c85e5b0e81"} Oct 29 22:54:16.266466 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.266403 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" podStartSLOduration=2.266389111 podStartE2EDuration="2.266389111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:16.2540792 +0000 UTC m=+653.369998807" watchObservedRunningTime="2026-10-29 22:54:16.266389111 +0000 UTC m=+653.382308794" Oct 29 22:54:16.318702 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.318659 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:16.320342 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.320319 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:54:16.320417 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.320382 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:54:16.859150 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.859117 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm"] Oct 29 22:54:16.863209 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.863181 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:16.876781 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.876759 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:54:16.876888 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.876824 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:54:16.884073 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.884046 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm"] Oct 29 22:54:16.956476 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.956432 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrc6z\" (UniqueName: \"kubernetes.io/projected/e3162219-21f7-434f-983b-61d2eeefed4b-kube-api-access-rrc6z\") pod \"maas-api-revoke-sub-0681236b-42jqm\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:16.956691 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.956631 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0681236b-42jqm\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:16.956691 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:16.956678 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0681236b-42jqm\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:17.057828 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.057787 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0681236b-42jqm\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:17.058040 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.057838 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0681236b-42jqm\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:17.058040 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.057979 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrc6z\" (UniqueName: \"kubernetes.io/projected/e3162219-21f7-434f-983b-61d2eeefed4b-kube-api-access-rrc6z\") pod \"maas-api-revoke-sub-0681236b-42jqm\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:17.058615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.058571 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0681236b-42jqm\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:17.060348 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.060324 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0681236b-42jqm\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:17.084421 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.084393 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrc6z\" (UniqueName: \"kubernetes.io/projected/e3162219-21f7-434f-983b-61d2eeefed4b-kube-api-access-rrc6z\") pod \"maas-api-revoke-sub-0681236b-42jqm\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:17.178089 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.178013 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:17.318639 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.318602 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:54:17.318965 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.318652 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:54:17.489070 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:17.489030 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm"] Oct 29 22:54:18.210200 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:18.210161 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" event={"ID":"e3162219-21f7-434f-983b-61d2eeefed4b","Type":"ContainerStarted","Data":"b0deaa057a3fbb7a5b626cc02ac201d1c5a480596bc44a31e80a7e54d431b369"} Oct 29 22:54:18.210346 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:18.210217 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" event={"ID":"e3162219-21f7-434f-983b-61d2eeefed4b","Type":"ContainerStarted","Data":"1db9552db312542d86ffd59c11636fc9f2e7bc425241043652375286ac012de6"} Oct 29 22:54:18.251446 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:18.251385 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" podStartSLOduration=2.251366959 podStartE2EDuration="2.251366959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:18.249097929 +0000 UTC m=+655.365017536" watchObservedRunningTime="2026-10-29 22:54:18.251366959 +0000 UTC m=+655.367286560" Oct 29 22:54:18.319018 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:18.318991 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:54:18.319365 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:18.319038 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:54:19.217671 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:19.217635 5736 generic.go:356] "Generic (PLEG): container finished" podID="e3162219-21f7-434f-983b-61d2eeefed4b" containerID="b0deaa057a3fbb7a5b626cc02ac201d1c5a480596bc44a31e80a7e54d431b369" exitCode=0 Oct 29 22:54:19.217933 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:19.217704 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" event={"ID":"e3162219-21f7-434f-983b-61d2eeefed4b","Type":"ContainerDied","Data":"b0deaa057a3fbb7a5b626cc02ac201d1c5a480596bc44a31e80a7e54d431b369"} Oct 29 22:54:19.318834 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:19.318793 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:54:19.318990 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:19.318864 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:54:20.319116 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.319082 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:54:20.319643 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.319155 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:54:20.396272 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.396245 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:20.498023 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.497942 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-openshift-ca\") pod \"e3162219-21f7-434f-983b-61d2eeefed4b\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " Oct 29 22:54:20.498023 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.498014 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3162219-21f7-434f-983b-61d2eeefed4b-kube-api-access-rrc6z\" (UniqueName: \"kubernetes.io/projected/e3162219-21f7-434f-983b-61d2eeefed4b-kube-api-access-rrc6z\") pod \"e3162219-21f7-434f-983b-61d2eeefed4b\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " Oct 29 22:54:20.498242 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.498070 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-xks-ca\") pod \"e3162219-21f7-434f-983b-61d2eeefed4b\" (UID: \"e3162219-21f7-434f-983b-61d2eeefed4b\") " Oct 29 22:54:20.498364 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.498344 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-openshift-ca" pod "e3162219-21f7-434f-983b-61d2eeefed4b" (UID: "e3162219-21f7-434f-983b-61d2eeefed4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:20.498514 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.498497 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:20.500204 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.500174 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3162219-21f7-434f-983b-61d2eeefed4b-kube-api-access-rrc6z" pod "e3162219-21f7-434f-983b-61d2eeefed4b" (UID: "e3162219-21f7-434f-983b-61d2eeefed4b"). InnerVolumeSpecName "kube-api-access-rrc6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:20.500323 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.500233 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-xks-ca" pod "e3162219-21f7-434f-983b-61d2eeefed4b" (UID: "e3162219-21f7-434f-983b-61d2eeefed4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:20.599525 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.599491 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrc6z\" (UniqueName: \"kubernetes.io/projected/e3162219-21f7-434f-983b-61d2eeefed4b-kube-api-access-rrc6z\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:20.599525 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:20.599519 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3162219-21f7-434f-983b-61d2eeefed4b-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:21.238604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:21.237948 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" event={"ID":"e3162219-21f7-434f-983b-61d2eeefed4b","Type":"ContainerDied","Data":"1db9552db312542d86ffd59c11636fc9f2e7bc425241043652375286ac012de6"} Oct 29 22:54:21.238604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:21.238014 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db9552db312542d86ffd59c11636fc9f2e7bc425241043652375286ac012de6" Oct 29 22:54:21.238604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:21.238082 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm" Oct 29 22:54:21.318565 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:21.318488 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:54:21.318565 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:21.318539 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:54:22.931695 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:22.931659 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:22.932188 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:22.932148 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:22.934951 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:22.934926 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 22:54:26.822976 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:54:26.822936 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:54:27.180547 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.180335 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8"] Oct 29 22:54:27.180841 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.180825 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e3162219-21f7-434f-983b-61d2eeefed4b" containerName="revoke-keys" Oct 29 22:54:27.180892 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.180843 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3162219-21f7-434f-983b-61d2eeefed4b" containerName="revoke-keys" Oct 29 22:54:27.180955 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.180946 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3162219-21f7-434f-983b-61d2eeefed4b" containerName="revoke-keys" Oct 29 22:54:27.185174 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.185154 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.198202 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.198180 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:54:27.208742 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.208720 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:54:27.211638 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.211610 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8"] Oct 29 22:54:27.269409 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.269383 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438d8f66-469p8\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.269534 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.269446 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438d8f66-469p8\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.269668 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.269647 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk7ss\" (UniqueName: \"kubernetes.io/projected/202674fe-a5f9-45cf-9641-84882f7ebe76-kube-api-access-qk7ss\") pod \"maas-api-revoke-sub-438d8f66-469p8\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.370455 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.370429 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk7ss\" (UniqueName: \"kubernetes.io/projected/202674fe-a5f9-45cf-9641-84882f7ebe76-kube-api-access-qk7ss\") pod \"maas-api-revoke-sub-438d8f66-469p8\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.370598 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.370478 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438d8f66-469p8\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.370598 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.370521 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438d8f66-469p8\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.371256 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.371237 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438d8f66-469p8\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.372915 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.372894 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438d8f66-469p8\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.391952 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.391929 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk7ss\" (UniqueName: \"kubernetes.io/projected/202674fe-a5f9-45cf-9641-84882f7ebe76-kube-api-access-qk7ss\") pod \"maas-api-revoke-sub-438d8f66-469p8\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.497603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.497562 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:27.796043 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:27.795967 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8"] Oct 29 22:54:27.799096 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:54:27.799067 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202674fe_a5f9_45cf_9641_84882f7ebe76.slice/crio-8a17751929f524aa82d1c7f9514127f88c80ccf7b9823d89dff99e67d6e4c078 WatchSource:0}: Error finding container 8a17751929f524aa82d1c7f9514127f88c80ccf7b9823d89dff99e67d6e4c078: Status 404 returned error can't find the container with id 8a17751929f524aa82d1c7f9514127f88c80ccf7b9823d89dff99e67d6e4c078 Oct 29 22:54:28.279611 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:28.279568 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" event={"ID":"202674fe-a5f9-45cf-9641-84882f7ebe76","Type":"ContainerStarted","Data":"25aa74a411f6e01d6f7dbb46faa506d34fcf6b77cb506a2996b2eb81c5e5fc65"} Oct 29 22:54:28.280033 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:28.279632 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" event={"ID":"202674fe-a5f9-45cf-9641-84882f7ebe76","Type":"ContainerStarted","Data":"8a17751929f524aa82d1c7f9514127f88c80ccf7b9823d89dff99e67d6e4c078"} Oct 29 22:54:28.318251 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:28.318204 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" podStartSLOduration=1.3181854450000001 podStartE2EDuration="1.318185445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:28.316245059 +0000 UTC m=+665.432164666" watchObservedRunningTime="2026-10-29 22:54:28.318185445 +0000 UTC m=+665.434105052" Oct 29 22:54:29.286629 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:29.286595 5736 generic.go:356] "Generic (PLEG): container finished" podID="202674fe-a5f9-45cf-9641-84882f7ebe76" containerID="25aa74a411f6e01d6f7dbb46faa506d34fcf6b77cb506a2996b2eb81c5e5fc65" exitCode=0 Oct 29 22:54:29.287002 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:29.286672 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" event={"ID":"202674fe-a5f9-45cf-9641-84882f7ebe76","Type":"ContainerDied","Data":"25aa74a411f6e01d6f7dbb46faa506d34fcf6b77cb506a2996b2eb81c5e5fc65"} Oct 29 22:54:30.433007 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.432980 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:30.504593 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.504556 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202674fe-a5f9-45cf-9641-84882f7ebe76-kube-api-access-qk7ss\" (UniqueName: \"kubernetes.io/projected/202674fe-a5f9-45cf-9641-84882f7ebe76-kube-api-access-qk7ss\") pod \"202674fe-a5f9-45cf-9641-84882f7ebe76\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " Oct 29 22:54:30.504731 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.504634 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-xks-ca\") pod \"202674fe-a5f9-45cf-9641-84882f7ebe76\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " Oct 29 22:54:30.504731 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.504681 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-openshift-ca\") pod \"202674fe-a5f9-45cf-9641-84882f7ebe76\" (UID: \"202674fe-a5f9-45cf-9641-84882f7ebe76\") " Oct 29 22:54:30.505109 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.505074 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-openshift-ca" pod "202674fe-a5f9-45cf-9641-84882f7ebe76" (UID: "202674fe-a5f9-45cf-9641-84882f7ebe76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:30.506648 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.506618 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202674fe-a5f9-45cf-9641-84882f7ebe76-kube-api-access-qk7ss" pod "202674fe-a5f9-45cf-9641-84882f7ebe76" (UID: "202674fe-a5f9-45cf-9641-84882f7ebe76"). InnerVolumeSpecName "kube-api-access-qk7ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:30.506757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.506719 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-xks-ca" pod "202674fe-a5f9-45cf-9641-84882f7ebe76" (UID: "202674fe-a5f9-45cf-9641-84882f7ebe76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:30.605840 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.605773 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:30.605840 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.605800 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202674fe-a5f9-45cf-9641-84882f7ebe76-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:30.605840 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:30.605815 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk7ss\" (UniqueName: \"kubernetes.io/projected/202674fe-a5f9-45cf-9641-84882f7ebe76-kube-api-access-qk7ss\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:54:31.299029 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:31.298953 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" Oct 29 22:54:31.299029 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:31.298999 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8" event={"ID":"202674fe-a5f9-45cf-9641-84882f7ebe76","Type":"ContainerDied","Data":"8a17751929f524aa82d1c7f9514127f88c80ccf7b9823d89dff99e67d6e4c078"} Oct 29 22:54:31.299231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:54:31.299056 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a17751929f524aa82d1c7f9514127f88c80ccf7b9823d89dff99e67d6e4c078" Oct 29 22:54:56.823241 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:54:56.823197 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:55:01.241235 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.241198 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484"] Oct 29 22:55:01.242054 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.242028 5736 cpu_manager.go:414] "containerMap: removing container" podUID="202674fe-a5f9-45cf-9641-84882f7ebe76" containerName="revoke-keys" Oct 29 22:55:01.242169 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.242056 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202674fe-a5f9-45cf-9641-84882f7ebe76" containerName="revoke-keys" Oct 29 22:55:01.242226 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.242218 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="202674fe-a5f9-45cf-9641-84882f7ebe76" containerName="revoke-keys" Oct 29 22:55:01.249292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.249267 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.268472 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.268445 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484"] Oct 29 22:55:01.272999 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.272963 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2077b4\"/\"distinct-w5-2077b4-kserve-self-signed-certs\"" Oct 29 22:55:01.333609 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.333557 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f2e1d-f61f-4da6-b374-537050d0c978-tls-certs\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.333609 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.333616 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-model-cache\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.333858 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.333689 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-dshm\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.333858 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.333773 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-tmp-dir\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.333858 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.333824 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clflq\" (UniqueName: \"kubernetes.io/projected/2d6f2e1d-f61f-4da6-b374-537050d0c978-kube-api-access-clflq\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.334016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.333951 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-home\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.435344 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.435297 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-tmp-dir\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.435551 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.435362 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clflq\" (UniqueName: \"kubernetes.io/projected/2d6f2e1d-f61f-4da6-b374-537050d0c978-kube-api-access-clflq\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.435551 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.435431 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-home\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.435695 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.435622 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f2e1d-f61f-4da6-b374-537050d0c978-tls-certs\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.435695 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.435663 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-model-cache\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.435776 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.435717 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-dshm\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.436437 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.436408 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-home\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.436621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.436487 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-tmp-dir\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.437023 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.436994 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-model-cache\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.438823 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.438798 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-dshm\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.439497 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.439474 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f2e1d-f61f-4da6-b374-537050d0c978-tls-certs\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.461326 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.461291 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clflq\" (UniqueName: \"kubernetes.io/projected/2d6f2e1d-f61f-4da6-b374-537050d0c978-kube-api-access-clflq\") pod \"distinct-w5-2077b4-kserve-7d4969db6-n5484\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.565456 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.565359 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:01.870490 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:01.870397 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484"] Oct 29 22:55:01.874444 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:55:01.874414 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d6f2e1d_f61f_4da6_b374_537050d0c978.slice/crio-dbd9cc48c793d70288728bf73b4426cbc3e6c21974c90e200e52bcd8a872eadb WatchSource:0}: Error finding container dbd9cc48c793d70288728bf73b4426cbc3e6c21974c90e200e52bcd8a872eadb: Status 404 returned error can't find the container with id dbd9cc48c793d70288728bf73b4426cbc3e6c21974c90e200e52bcd8a872eadb Oct 29 22:55:02.486592 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:02.486553 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" event={"ID":"2d6f2e1d-f61f-4da6-b374-537050d0c978","Type":"ContainerStarted","Data":"a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc"} Oct 29 22:55:02.486983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:02.486625 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" event={"ID":"2d6f2e1d-f61f-4da6-b374-537050d0c978","Type":"ContainerStarted","Data":"dbd9cc48c793d70288728bf73b4426cbc3e6c21974c90e200e52bcd8a872eadb"} Oct 29 22:55:02.524397 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:02.524341 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podStartSLOduration=1.5243267230000002 podStartE2EDuration="1.524326723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:02.52299766 +0000 UTC m=+699.638917265" watchObservedRunningTime="2026-10-29 22:55:02.524326723 +0000 UTC m=+699.640246329" Oct 29 22:55:02.565833 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:02.565804 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:03.405871 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.405834 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc"] Oct 29 22:55:03.409923 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.409906 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.422639 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.422614 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:55:03.432733 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.432707 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:55:03.434916 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.434881 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc"] Oct 29 22:55:03.560479 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.560447 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d5d776-87jlc\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.560897 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.560659 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbfv\" (UniqueName: \"kubernetes.io/projected/c9c74229-1968-40a8-b6a8-017e59e03867-kube-api-access-ffbfv\") pod \"maas-api-revoke-sub-84d5d776-87jlc\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.560897 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.560828 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d5d776-87jlc\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.662699 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.662613 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbfv\" (UniqueName: \"kubernetes.io/projected/c9c74229-1968-40a8-b6a8-017e59e03867-kube-api-access-ffbfv\") pod \"maas-api-revoke-sub-84d5d776-87jlc\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.662869 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.662723 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d5d776-87jlc\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.662869 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.662812 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d5d776-87jlc\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.663449 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.663424 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d5d776-87jlc\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.665155 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.665134 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d5d776-87jlc\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.683927 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.683902 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbfv\" (UniqueName: \"kubernetes.io/projected/c9c74229-1968-40a8-b6a8-017e59e03867-kube-api-access-ffbfv\") pod \"maas-api-revoke-sub-84d5d776-87jlc\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:03.723982 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:03.723954 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:04.041353 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:04.041305 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc"] Oct 29 22:55:04.051295 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:55:04.051261 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c74229_1968_40a8_b6a8_017e59e03867.slice/crio-f4b223b2b895544cb905411930682738be8e8895def454b2e5fdda0f4e4015d9 WatchSource:0}: Error finding container f4b223b2b895544cb905411930682738be8e8895def454b2e5fdda0f4e4015d9: Status 404 returned error can't find the container with id f4b223b2b895544cb905411930682738be8e8895def454b2e5fdda0f4e4015d9 Oct 29 22:55:04.500665 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:04.500627 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" event={"ID":"c9c74229-1968-40a8-b6a8-017e59e03867","Type":"ContainerStarted","Data":"c8029e3ba17d33782d196ebe68932f27d89fe2c6312fecb5f25249201c8226be"} Oct 29 22:55:04.500864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:04.500686 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" event={"ID":"c9c74229-1968-40a8-b6a8-017e59e03867","Type":"ContainerStarted","Data":"f4b223b2b895544cb905411930682738be8e8895def454b2e5fdda0f4e4015d9"} Oct 29 22:55:04.537399 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:04.537338 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" podStartSLOduration=1.537319546 podStartE2EDuration="1.537319546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:04.535422957 +0000 UTC m=+701.651342563" watchObservedRunningTime="2026-10-29 22:55:04.537319546 +0000 UTC m=+701.653239153" Oct 29 22:55:05.508886 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:05.508854 5736 generic.go:356] "Generic (PLEG): container finished" podID="c9c74229-1968-40a8-b6a8-017e59e03867" containerID="c8029e3ba17d33782d196ebe68932f27d89fe2c6312fecb5f25249201c8226be" exitCode=0 Oct 29 22:55:05.509310 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:05.508903 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" event={"ID":"c9c74229-1968-40a8-b6a8-017e59e03867","Type":"ContainerDied","Data":"c8029e3ba17d33782d196ebe68932f27d89fe2c6312fecb5f25249201c8226be"} Oct 29 22:55:06.655497 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.655473 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:06.760677 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.760642 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz"] Oct 29 22:55:06.761475 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.761327 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c9c74229-1968-40a8-b6a8-017e59e03867" containerName="revoke-keys" Oct 29 22:55:06.761475 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.761345 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c74229-1968-40a8-b6a8-017e59e03867" containerName="revoke-keys" Oct 29 22:55:06.761475 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.761455 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c74229-1968-40a8-b6a8-017e59e03867" containerName="revoke-keys" Oct 29 22:55:06.764524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.764502 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:06.777260 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.777236 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"distinct-w4-ed8218-kserve-self-signed-certs\"" Oct 29 22:55:06.785769 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.785743 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz"] Oct 29 22:55:06.798362 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.798341 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-xks-ca\") pod \"c9c74229-1968-40a8-b6a8-017e59e03867\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " Oct 29 22:55:06.798603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.798565 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c74229-1968-40a8-b6a8-017e59e03867-kube-api-access-ffbfv\" (UniqueName: \"kubernetes.io/projected/c9c74229-1968-40a8-b6a8-017e59e03867-kube-api-access-ffbfv\") pod \"c9c74229-1968-40a8-b6a8-017e59e03867\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " Oct 29 22:55:06.798723 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.798640 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-openshift-ca\") pod \"c9c74229-1968-40a8-b6a8-017e59e03867\" (UID: \"c9c74229-1968-40a8-b6a8-017e59e03867\") " Oct 29 22:55:06.799048 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.799020 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-openshift-ca" pod "c9c74229-1968-40a8-b6a8-017e59e03867" (UID: "c9c74229-1968-40a8-b6a8-017e59e03867"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:06.801007 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.800985 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-xks-ca" pod "c9c74229-1968-40a8-b6a8-017e59e03867" (UID: "c9c74229-1968-40a8-b6a8-017e59e03867"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:06.806680 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.806657 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c74229-1968-40a8-b6a8-017e59e03867-kube-api-access-ffbfv" pod "c9c74229-1968-40a8-b6a8-017e59e03867" (UID: "c9c74229-1968-40a8-b6a8-017e59e03867"). InnerVolumeSpecName "kube-api-access-ffbfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:06.899996 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.899968 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-home\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:06.899996 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.899999 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tls-certs\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:06.900240 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.900032 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-model-cache\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:06.900240 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.900212 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tmp-dir\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:06.900363 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.900327 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-dshm\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:06.900418 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.900359 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt8dn\" (UniqueName: \"kubernetes.io/projected/e8086c08-bc06-46cf-bd5c-09f54c5b9700-kube-api-access-zt8dn\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:06.900474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.900440 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffbfv\" (UniqueName: \"kubernetes.io/projected/c9c74229-1968-40a8-b6a8-017e59e03867-kube-api-access-ffbfv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:55:06.900529 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.900481 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:55:06.900529 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:06.900504 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c74229-1968-40a8-b6a8-017e59e03867-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:55:07.002874 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.002833 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-dshm\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.003037 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.002906 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt8dn\" (UniqueName: \"kubernetes.io/projected/e8086c08-bc06-46cf-bd5c-09f54c5b9700-kube-api-access-zt8dn\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.003037 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.002980 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-home\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.003037 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.003006 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tls-certs\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.003183 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.003054 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-model-cache\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.003183 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.003132 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tmp-dir\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.003756 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.003726 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tmp-dir\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.003895 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.003807 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-home\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.004378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.004354 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-model-cache\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.006561 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.006515 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-dshm\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.019903 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.008222 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tls-certs\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.025995 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.025935 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt8dn\" (UniqueName: \"kubernetes.io/projected/e8086c08-bc06-46cf-bd5c-09f54c5b9700-kube-api-access-zt8dn\") pod \"distinct-w4-ed8218-kserve-679d796d67-km4zz\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.078677 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.078639 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:07.387036 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.386950 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz"] Oct 29 22:55:07.390502 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:55:07.390461 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8086c08_bc06_46cf_bd5c_09f54c5b9700.slice/crio-9eb0d0fe1a7b56f25ecfaa31e35eec2c326fbb13ddefc5ef3dc0d766b6765211 WatchSource:0}: Error finding container 9eb0d0fe1a7b56f25ecfaa31e35eec2c326fbb13ddefc5ef3dc0d766b6765211: Status 404 returned error can't find the container with id 9eb0d0fe1a7b56f25ecfaa31e35eec2c326fbb13ddefc5ef3dc0d766b6765211 Oct 29 22:55:07.521689 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.521661 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" Oct 29 22:55:07.521846 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.521661 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc" event={"ID":"c9c74229-1968-40a8-b6a8-017e59e03867","Type":"ContainerDied","Data":"f4b223b2b895544cb905411930682738be8e8895def454b2e5fdda0f4e4015d9"} Oct 29 22:55:07.521846 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.521782 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b223b2b895544cb905411930682738be8e8895def454b2e5fdda0f4e4015d9" Oct 29 22:55:07.523445 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.523410 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" event={"ID":"e8086c08-bc06-46cf-bd5c-09f54c5b9700","Type":"ContainerStarted","Data":"5f71b1edfe1200041b46f42806390cfaf1629f1ff867c90ac351256802a55135"} Oct 29 22:55:07.523591 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.523448 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" event={"ID":"e8086c08-bc06-46cf-bd5c-09f54c5b9700","Type":"ContainerStarted","Data":"9eb0d0fe1a7b56f25ecfaa31e35eec2c326fbb13ddefc5ef3dc0d766b6765211"} Oct 29 22:55:07.562421 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:07.562374 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podStartSLOduration=1.562361504 podStartE2EDuration="1.562361504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:07.560296543 +0000 UTC m=+704.676216148" watchObservedRunningTime="2026-10-29 22:55:07.562361504 +0000 UTC m=+704.678281110" Oct 29 22:55:08.079670 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.079640 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:08.335126 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.335034 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb"] Oct 29 22:55:08.338946 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.338923 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.361509 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.361479 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:55:08.361735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.361507 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:55:08.364961 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.364938 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb"] Oct 29 22:55:08.419282 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.419248 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6gw\" (UniqueName: \"kubernetes.io/projected/985a621f-26f7-40c1-a406-551845957ee0-kube-api-access-dw6gw\") pod \"maas-api-revoke-sub-13fffb5d-hjsmb\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.419393 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.419330 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985a621f-26f7-40c1-a406-551845957ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13fffb5d-hjsmb\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.419510 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.419479 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985a621f-26f7-40c1-a406-551845957ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13fffb5d-hjsmb\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.520130 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.520105 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985a621f-26f7-40c1-a406-551845957ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13fffb5d-hjsmb\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.520247 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.520180 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6gw\" (UniqueName: \"kubernetes.io/projected/985a621f-26f7-40c1-a406-551845957ee0-kube-api-access-dw6gw\") pod \"maas-api-revoke-sub-13fffb5d-hjsmb\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.520247 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.520222 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985a621f-26f7-40c1-a406-551845957ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13fffb5d-hjsmb\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.520749 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.520730 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985a621f-26f7-40c1-a406-551845957ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13fffb5d-hjsmb\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.522473 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.522453 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985a621f-26f7-40c1-a406-551845957ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13fffb5d-hjsmb\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.541774 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.541746 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6gw\" (UniqueName: \"kubernetes.io/projected/985a621f-26f7-40c1-a406-551845957ee0-kube-api-access-dw6gw\") pod \"maas-api-revoke-sub-13fffb5d-hjsmb\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.652471 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.652405 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:08.953402 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:08.953174 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb"] Oct 29 22:55:08.956435 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:55:08.956402 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985a621f_26f7_40c1_a406_551845957ee0.slice/crio-9f6ea015635668b4e1fef72d097d6ffdba4180bbcb64870a5106d06d2666b959 WatchSource:0}: Error finding container 9f6ea015635668b4e1fef72d097d6ffdba4180bbcb64870a5106d06d2666b959: Status 404 returned error can't find the container with id 9f6ea015635668b4e1fef72d097d6ffdba4180bbcb64870a5106d06d2666b959 Oct 29 22:55:09.539370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:09.539292 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" event={"ID":"985a621f-26f7-40c1-a406-551845957ee0","Type":"ContainerStarted","Data":"4148fafc2e15d877c3e7d116e2ec483d438421bf76c132a39e4ce4c1d5110baf"} Oct 29 22:55:09.539370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:09.539347 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" event={"ID":"985a621f-26f7-40c1-a406-551845957ee0","Type":"ContainerStarted","Data":"9f6ea015635668b4e1fef72d097d6ffdba4180bbcb64870a5106d06d2666b959"} Oct 29 22:55:09.576451 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:09.576406 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" podStartSLOduration=1.57639393 podStartE2EDuration="1.57639393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:09.575274178 +0000 UTC m=+706.691193785" watchObservedRunningTime="2026-10-29 22:55:09.57639393 +0000 UTC m=+706.692313536" Oct 29 22:55:10.547053 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:10.547012 5736 generic.go:356] "Generic (PLEG): container finished" podID="985a621f-26f7-40c1-a406-551845957ee0" containerID="4148fafc2e15d877c3e7d116e2ec483d438421bf76c132a39e4ce4c1d5110baf" exitCode=0 Oct 29 22:55:10.547450 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:10.547072 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" event={"ID":"985a621f-26f7-40c1-a406-551845957ee0","Type":"ContainerDied","Data":"4148fafc2e15d877c3e7d116e2ec483d438421bf76c132a39e4ce4c1d5110baf"} Oct 29 22:55:11.565552 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.565511 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:11.581948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.581920 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:11.689358 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.689336 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:11.857156 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.857078 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985a621f-26f7-40c1-a406-551845957ee0-kube-api-access-dw6gw\" (UniqueName: \"kubernetes.io/projected/985a621f-26f7-40c1-a406-551845957ee0-kube-api-access-dw6gw\") pod \"985a621f-26f7-40c1-a406-551845957ee0\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " Oct 29 22:55:11.857289 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.857219 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985a621f-26f7-40c1-a406-551845957ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985a621f-26f7-40c1-a406-551845957ee0-maas-api-openshift-ca\") pod \"985a621f-26f7-40c1-a406-551845957ee0\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " Oct 29 22:55:11.857354 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.857286 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/985a621f-26f7-40c1-a406-551845957ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985a621f-26f7-40c1-a406-551845957ee0-maas-api-xks-ca\") pod \"985a621f-26f7-40c1-a406-551845957ee0\" (UID: \"985a621f-26f7-40c1-a406-551845957ee0\") " Oct 29 22:55:11.857672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.857640 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985a621f-26f7-40c1-a406-551845957ee0-maas-api-openshift-ca" pod "985a621f-26f7-40c1-a406-551845957ee0" (UID: "985a621f-26f7-40c1-a406-551845957ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:11.859170 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.859141 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985a621f-26f7-40c1-a406-551845957ee0-kube-api-access-dw6gw" pod "985a621f-26f7-40c1-a406-551845957ee0" (UID: "985a621f-26f7-40c1-a406-551845957ee0"). InnerVolumeSpecName "kube-api-access-dw6gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:11.859517 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.859492 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985a621f-26f7-40c1-a406-551845957ee0-maas-api-xks-ca" pod "985a621f-26f7-40c1-a406-551845957ee0" (UID: "985a621f-26f7-40c1-a406-551845957ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:11.958513 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.958487 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985a621f-26f7-40c1-a406-551845957ee0-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:55:11.958513 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.958512 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985a621f-26f7-40c1-a406-551845957ee0-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:55:11.958675 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:11.958523 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6gw\" (UniqueName: \"kubernetes.io/projected/985a621f-26f7-40c1-a406-551845957ee0-kube-api-access-dw6gw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:55:12.559412 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:12.559378 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" event={"ID":"985a621f-26f7-40c1-a406-551845957ee0","Type":"ContainerDied","Data":"9f6ea015635668b4e1fef72d097d6ffdba4180bbcb64870a5106d06d2666b959"} Oct 29 22:55:12.559603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:12.559430 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f6ea015635668b4e1fef72d097d6ffdba4180bbcb64870a5106d06d2666b959" Oct 29 22:55:12.559603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:12.559466 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb" Oct 29 22:55:12.570960 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:12.570937 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:55:17.079465 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:17.079430 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:17.092043 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:17.092021 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:17.599281 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:17.599251 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:55:26.822962 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:55:26.822917 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:55:29.583827 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.583787 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p"] Oct 29 22:55:29.584831 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.584505 5736 cpu_manager.go:414] "containerMap: removing container" podUID="985a621f-26f7-40c1-a406-551845957ee0" containerName="revoke-keys" Oct 29 22:55:29.584831 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.584526 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985a621f-26f7-40c1-a406-551845957ee0" containerName="revoke-keys" Oct 29 22:55:29.584831 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.584685 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="985a621f-26f7-40c1-a406-551845957ee0" containerName="revoke-keys" Oct 29 22:55:29.588990 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.588968 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.611925 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.611895 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1ee571\"/\"distinct-w6-1ee571-kserve-self-signed-certs\"" Oct 29 22:55:29.615624 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.615562 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p"] Oct 29 22:55:29.653098 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.653071 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tmp-dir\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.653242 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.653218 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-home\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.653290 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.653264 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-dshm\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.653370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.653350 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-model-cache\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.653506 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.653492 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tls-certs\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.653545 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.653516 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6np\" (UniqueName: \"kubernetes.io/projected/e6a5318b-2f1c-4171-8242-2a9c94c517e4-kube-api-access-xj6np\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.755004 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.754958 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-home\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.755004 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.755007 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-dshm\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.755248 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.755035 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-model-cache\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.755248 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.755165 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tls-certs\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.755248 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.755194 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6np\" (UniqueName: \"kubernetes.io/projected/e6a5318b-2f1c-4171-8242-2a9c94c517e4-kube-api-access-xj6np\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.755438 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.755276 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tmp-dir\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.755542 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.755514 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-model-cache\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.755666 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.755645 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tmp-dir\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.755766 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.755744 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-home\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.757478 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.757424 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-dshm\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.757775 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.757751 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tls-certs\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.782656 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.779274 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6np\" (UniqueName: \"kubernetes.io/projected/e6a5318b-2f1c-4171-8242-2a9c94c517e4-kube-api-access-xj6np\") pod \"distinct-w6-1ee571-kserve-55bbf744c9-b6b6p\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:29.904517 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:29.904433 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:30.208882 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:30.208801 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p"] Oct 29 22:55:30.212017 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:55:30.211991 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6a5318b_2f1c_4171_8242_2a9c94c517e4.slice/crio-c95d327d5cb866f381b5c29238070596999e1ac091c105933c42ae18c6a3237b WatchSource:0}: Error finding container c95d327d5cb866f381b5c29238070596999e1ac091c105933c42ae18c6a3237b: Status 404 returned error can't find the container with id c95d327d5cb866f381b5c29238070596999e1ac091c105933c42ae18c6a3237b Oct 29 22:55:30.670231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:30.670191 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" event={"ID":"e6a5318b-2f1c-4171-8242-2a9c94c517e4","Type":"ContainerStarted","Data":"4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a"} Oct 29 22:55:30.670670 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:30.670247 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" event={"ID":"e6a5318b-2f1c-4171-8242-2a9c94c517e4","Type":"ContainerStarted","Data":"c95d327d5cb866f381b5c29238070596999e1ac091c105933c42ae18c6a3237b"} Oct 29 22:55:30.707909 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:30.707856 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podStartSLOduration=1.7078434489999998 podStartE2EDuration="1.707843449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:30.706984762 +0000 UTC m=+727.822904369" watchObservedRunningTime="2026-10-29 22:55:30.707843449 +0000 UTC m=+727.823763054" Oct 29 22:55:30.905565 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:30.905537 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:39.904878 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:39.904800 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:39.917072 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:39.917053 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:40.746271 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:40.746237 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:55:51.748847 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.748810 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv"] Oct 29 22:55:51.753612 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.753572 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.777017 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.776989 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"distinct-2-w3-32c192-kserve-self-signed-certs\"" Oct 29 22:55:51.779379 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.779354 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv"] Oct 29 22:55:51.797315 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.797285 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-home\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.797454 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.797318 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tmp-dir\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.797454 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.797435 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkt2n\" (UniqueName: \"kubernetes.io/projected/1ea31536-fd5e-489b-8092-8d9a8cb816cd-kube-api-access-bkt2n\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.797615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.797552 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-dshm\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.797615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.797598 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tls-certs\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.797739 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.797684 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-model-cache\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.899057 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.899020 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-home\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.899057 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.899067 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tmp-dir\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.899292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.899119 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkt2n\" (UniqueName: \"kubernetes.io/projected/1ea31536-fd5e-489b-8092-8d9a8cb816cd-kube-api-access-bkt2n\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.899292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.899211 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-dshm\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.899292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.899243 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tls-certs\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.899292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.899290 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-model-cache\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.899509 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.899451 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-home\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.899662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.899639 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tmp-dir\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.899748 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.899725 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-model-cache\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.902043 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.902009 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-dshm\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.902176 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.902017 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tls-certs\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:51.922948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:51.922918 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkt2n\" (UniqueName: \"kubernetes.io/projected/1ea31536-fd5e-489b-8092-8d9a8cb816cd-kube-api-access-bkt2n\") pod \"distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:52.068946 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:52.068853 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:52.373838 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:52.373764 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv"] Oct 29 22:55:52.378014 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:55:52.377983 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ea31536_fd5e_489b_8092_8d9a8cb816cd.slice/crio-293a6d518114fc1167bdd902379c25215cb44c2993d98a46e7580ecc2cf1f33f WatchSource:0}: Error finding container 293a6d518114fc1167bdd902379c25215cb44c2993d98a46e7580ecc2cf1f33f: Status 404 returned error can't find the container with id 293a6d518114fc1167bdd902379c25215cb44c2993d98a46e7580ecc2cf1f33f Oct 29 22:55:52.803974 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:52.803937 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" event={"ID":"1ea31536-fd5e-489b-8092-8d9a8cb816cd","Type":"ContainerStarted","Data":"ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c"} Oct 29 22:55:52.804366 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:52.803989 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" event={"ID":"1ea31536-fd5e-489b-8092-8d9a8cb816cd","Type":"ContainerStarted","Data":"293a6d518114fc1167bdd902379c25215cb44c2993d98a46e7580ecc2cf1f33f"} Oct 29 22:55:52.841365 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:52.841322 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podStartSLOduration=1.841309679 podStartE2EDuration="1.841309679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:52.840822796 +0000 UTC m=+749.956742404" watchObservedRunningTime="2026-10-29 22:55:52.841309679 +0000 UTC m=+749.957229284" Oct 29 22:55:53.069628 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:53.069534 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:55:56.390096 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.390059 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh"] Oct 29 22:55:56.394097 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.394076 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.416769 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.416742 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"distinct-2-w4-ed8218-kserve-self-signed-certs\"" Oct 29 22:55:56.419269 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.419244 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh"] Oct 29 22:55:56.446946 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.446914 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-model-cache\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.447116 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.447028 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tls-certs\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.447116 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.447074 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229kf\" (UniqueName: \"kubernetes.io/projected/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-kube-api-access-229kf\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.447209 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.447192 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-dshm\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.447249 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.447220 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-home\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.447249 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.447241 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tmp-dir\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.548197 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.548156 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tls-certs\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.548397 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.548205 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229kf\" (UniqueName: \"kubernetes.io/projected/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-kube-api-access-229kf\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.548397 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.548341 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-dshm\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.548397 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.548373 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-home\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.548573 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.548397 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tmp-dir\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.548799 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.548764 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-model-cache\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.548799 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.548786 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-home\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.549007 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.548857 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tmp-dir\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.549080 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.549063 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-model-cache\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.550821 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.550793 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-dshm\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.551192 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.551174 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tls-certs\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.575602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.571345 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-229kf\" (UniqueName: \"kubernetes.io/projected/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-kube-api-access-229kf\") pod \"distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.708120 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:56.708034 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:55:56.822989 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:55:56.822951 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:55:57.017851 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:57.017816 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh"] Oct 29 22:55:57.021098 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:55:57.021070 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadbe7c2a_9f80_4dac_96b0_f8734a3759e8.slice/crio-b43be0b03dc7a6bf07464e8f1288c349f821dd59b021e2a1bb9ce0eaf60e9eb3 WatchSource:0}: Error finding container b43be0b03dc7a6bf07464e8f1288c349f821dd59b021e2a1bb9ce0eaf60e9eb3: Status 404 returned error can't find the container with id b43be0b03dc7a6bf07464e8f1288c349f821dd59b021e2a1bb9ce0eaf60e9eb3 Oct 29 22:55:57.833400 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:57.833364 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" event={"ID":"adbe7c2a-9f80-4dac-96b0-f8734a3759e8","Type":"ContainerStarted","Data":"3e71961977144b59ea9a6169e5749c2eec9f36078cd5b6710f8be97602141dfa"} Oct 29 22:55:57.833865 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:57.833411 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" event={"ID":"adbe7c2a-9f80-4dac-96b0-f8734a3759e8","Type":"ContainerStarted","Data":"b43be0b03dc7a6bf07464e8f1288c349f821dd59b021e2a1bb9ce0eaf60e9eb3"} Oct 29 22:55:57.871170 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:57.871125 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podStartSLOduration=1.87111259 podStartE2EDuration="1.87111259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:57.87062171 +0000 UTC m=+754.986541315" watchObservedRunningTime="2026-10-29 22:55:57.87111259 +0000 UTC m=+754.987032195" Oct 29 22:55:58.709173 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:55:58.709139 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:56:02.069366 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:02.069336 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:56:02.082166 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:02.082137 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:56:02.873623 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:02.873594 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:56:06.708804 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:06.708770 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:56:06.721122 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:06.721093 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:56:06.897709 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:06.897677 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:56:12.994060 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:12.994023 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp"] Oct 29 22:56:13.000206 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.000178 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.014445 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.014418 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-1ee571-openshift-default-dockercfg-bmxwc\"" Oct 29 22:56:13.027234 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.027194 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp"] Oct 29 22:56:13.133936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.133905 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.134089 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.133945 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttgp\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-kube-api-access-vttgp\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.134089 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.134025 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.134089 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.134074 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.134193 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.134161 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.134227 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.134215 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.134260 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.134252 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.134389 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.134369 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.134473 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.134407 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.235707 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.235673 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.235874 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.235724 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttgp\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-kube-api-access-vttgp\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.235874 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.235754 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.235874 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.235813 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.235874 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.235850 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.236109 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.235901 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.236109 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.235957 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.236109 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.236006 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.236109 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.236024 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.236109 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.236056 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.236365 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.236249 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.236492 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.236467 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.236492 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.236487 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.236636 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.236607 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.238496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.238469 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.238657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.238636 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.257358 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.257302 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.257436 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.257400 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttgp\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-kube-api-access-vttgp\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.327270 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.327241 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:13.658974 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.658894 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp"] Oct 29 22:56:13.662089 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:13.662052 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5432e2_4ef8_41ce_9f7c_8e01ac9bcca4.slice/crio-b12331ee990f9bbaa9d8111ad87763e90c0fced81d043267ee95897e7ea539b4 WatchSource:0}: Error finding container b12331ee990f9bbaa9d8111ad87763e90c0fced81d043267ee95897e7ea539b4: Status 404 returned error can't find the container with id b12331ee990f9bbaa9d8111ad87763e90c0fced81d043267ee95897e7ea539b4 Oct 29 22:56:13.664843 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.664807 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:13.664943 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.664890 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:13.664943 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.664934 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:13.934388 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.934301 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" event={"ID":"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4","Type":"ContainerStarted","Data":"139b627c7e0cea3285bafc0b9fd5c694ffe48710ea2e717f6c66540ccf16d15a"} Oct 29 22:56:13.934388 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.934364 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" event={"ID":"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4","Type":"ContainerStarted","Data":"b12331ee990f9bbaa9d8111ad87763e90c0fced81d043267ee95897e7ea539b4"} Oct 29 22:56:13.980790 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:13.980725 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" podStartSLOduration=1.98070685 podStartE2EDuration="1.98070685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:13.97580319 +0000 UTC m=+771.091722795" watchObservedRunningTime="2026-10-29 22:56:13.98070685 +0000 UTC m=+771.096626457" Oct 29 22:56:14.328114 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:14.328083 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:14.329673 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:14.329645 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:56:14.329780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:14.329694 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:56:15.328222 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:15.328187 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:56:15.328736 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:15.328241 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:56:16.328132 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:16.328095 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:56:16.328311 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:16.328147 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:56:17.328208 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:17.328171 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:56:17.328378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:17.328229 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:56:18.328475 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:18.328440 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:56:18.328891 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:18.328494 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:56:19.328485 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:19.328451 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:56:19.328900 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:19.328502 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:56:20.327944 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:20.327903 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:56:20.328139 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:20.327966 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:56:21.331821 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.331785 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:21.332191 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.332079 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:21.332768 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.332752 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 22:56:21.627370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.627289 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995"] Oct 29 22:56:21.631198 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.631176 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.644005 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.643983 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ed8218-openshift-default-dockercfg-x28tf\"" Oct 29 22:56:21.656192 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.656167 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995"] Oct 29 22:56:21.743849 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.743819 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.743988 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.743873 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnxm\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-kube-api-access-2dnxm\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.744078 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.744022 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.744146 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.744088 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.744146 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.744122 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.744232 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.744207 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.744232 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.744228 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.744361 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.744248 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9778-d69a-4264-bcd5-1d13dfb17640-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.744361 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.744275 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.845597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.845548 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnxm\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-kube-api-access-2dnxm\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.845772 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.845670 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.845772 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.845728 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.845772 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.845758 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.845948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.845828 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.845948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.845859 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.845948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.845890 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9778-d69a-4264-bcd5-1d13dfb17640-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.845948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.845938 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.846152 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.846021 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.846214 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.846172 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.846300 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.846276 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.846558 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.846532 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.846659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.846569 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.846910 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.846888 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9778-d69a-4264-bcd5-1d13dfb17640-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.848412 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.848389 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.848747 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.848729 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.868426 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.868400 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.868620 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.868569 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnxm\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-kube-api-access-2dnxm\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rb995\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:21.944881 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:21.944809 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:22.267596 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:22.267548 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995"] Oct 29 22:56:22.271048 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:22.271011 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210b9778_d69a_4264_bcd5_1d13dfb17640.slice/crio-d745ee6c828b105f7ca73275e9e554d1c2a9f8e2f839bc0cbaac982a315facd6 WatchSource:0}: Error finding container d745ee6c828b105f7ca73275e9e554d1c2a9f8e2f839bc0cbaac982a315facd6: Status 404 returned error can't find the container with id d745ee6c828b105f7ca73275e9e554d1c2a9f8e2f839bc0cbaac982a315facd6 Oct 29 22:56:22.273482 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:22.273451 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:22.273566 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:22.273511 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:22.273566 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:22.273546 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:23.022401 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:23.022359 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" event={"ID":"210b9778-d69a-4264-bcd5-1d13dfb17640","Type":"ContainerStarted","Data":"3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a"} Oct 29 22:56:23.022881 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:23.022421 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" event={"ID":"210b9778-d69a-4264-bcd5-1d13dfb17640","Type":"ContainerStarted","Data":"d745ee6c828b105f7ca73275e9e554d1c2a9f8e2f839bc0cbaac982a315facd6"} Oct 29 22:56:23.074651 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:23.074604 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podStartSLOduration=2.074566137 podStartE2EDuration="2.074566137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:23.072668042 +0000 UTC m=+780.188587648" watchObservedRunningTime="2026-10-29 22:56:23.074566137 +0000 UTC m=+780.190485743" Oct 29 22:56:23.952601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:23.952241 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:23.952601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:23.952376 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:56:23.952601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:23.952431 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:56:24.555889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.555852 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf"] Oct 29 22:56:24.562827 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.562806 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.578717 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.578691 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.578848 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.578738 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870c69cc-31f5-45a3-84a3-113138e2c8ee-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.578917 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.578859 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.578917 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.578904 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.579019 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.578948 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.579019 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.578975 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.579107 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.579020 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.579107 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.579075 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz5kg\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-kube-api-access-tz5kg\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.579178 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.579123 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.580518 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.580500 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2077b4-openshift-default-dockercfg-4kftx\"" Oct 29 22:56:24.594327 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.594306 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf"] Oct 29 22:56:24.680304 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680264 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680320 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870c69cc-31f5-45a3-84a3-113138e2c8ee-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680371 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680413 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680454 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680724 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680475 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680724 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680506 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680724 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680541 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz5kg\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-kube-api-access-tz5kg\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680724 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680568 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680933 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680859 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.680995 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.680940 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.681100 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.681001 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.681168 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.681122 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.681168 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.681149 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870c69cc-31f5-45a3-84a3-113138e2c8ee-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.682796 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.682778 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.682994 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.682976 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.701708 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.701686 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.701824 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.701713 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz5kg\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-kube-api-access-tz5kg\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.877211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.877123 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:24.961860 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.955005 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:56:24.961860 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:24.955059 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:56:25.221621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:25.221569 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf"] Oct 29 22:56:25.224933 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:25.224905 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870c69cc_31f5_45a3_84a3_113138e2c8ee.slice/crio-ed11f61de045780c103a1cca184886f92147fe4c9458dd822d7ae93ada945d87 WatchSource:0}: Error finding container ed11f61de045780c103a1cca184886f92147fe4c9458dd822d7ae93ada945d87: Status 404 returned error can't find the container with id ed11f61de045780c103a1cca184886f92147fe4c9458dd822d7ae93ada945d87 Oct 29 22:56:25.227476 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:25.227443 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:25.227541 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:25.227509 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:25.227541 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:25.227537 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:25.953603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:25.952905 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:56:25.953603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:25.952976 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:56:26.069513 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:26.069479 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" event={"ID":"870c69cc-31f5-45a3-84a3-113138e2c8ee","Type":"ContainerStarted","Data":"60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e"} Oct 29 22:56:26.069690 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:26.069533 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" event={"ID":"870c69cc-31f5-45a3-84a3-113138e2c8ee","Type":"ContainerStarted","Data":"ed11f61de045780c103a1cca184886f92147fe4c9458dd822d7ae93ada945d87"} Oct 29 22:56:26.123107 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:26.123057 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" podStartSLOduration=2.123040645 podStartE2EDuration="2.123040645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:26.12238154 +0000 UTC m=+783.238301151" watchObservedRunningTime="2026-10-29 22:56:26.123040645 +0000 UTC m=+783.238960250" Oct 29 22:56:26.824360 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:56:26.824319 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:56:26.881860 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:26.881742 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:26.883703 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:26.883676 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:56:26.883806 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:26.883754 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:56:26.965842 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:26.965803 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:56:26.967752 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:26.967717 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:56:27.891604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:27.877946 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:56:27.891604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:27.878016 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:56:27.955604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:27.954974 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:56:27.955604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:27.955037 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:56:28.877470 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:28.877433 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:56:28.877865 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:28.877487 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:56:28.951603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:28.950943 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:56:28.951603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:28.951009 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:56:29.881916 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:29.881882 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:56:29.882544 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:29.882513 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:56:30.725601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.724722 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:30.725601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.725319 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:30.727573 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.726262 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 22:56:30.755672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.755625 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv"] Oct 29 22:56:30.778751 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.778704 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.795077 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.795042 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-32c192-openshift-default-dockercfg-m9mqd\"" Oct 29 22:56:30.858596 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.858535 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv"] Oct 29 22:56:30.925447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.925397 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89a6f30-fbab-4350-b331-aa0f2a19b656-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.926161 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.926139 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.926323 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.926306 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.926473 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.926453 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.926770 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.926752 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.926968 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.926952 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.927227 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.927212 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.927416 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.927402 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cm46\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-kube-api-access-6cm46\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.927540 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.927527 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:30.928521 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.928500 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:56:30.928831 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:30.928808 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:56:31.062149 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.062048 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.062407 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.062385 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.062549 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.062535 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.067603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.062772 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.067851 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.067829 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.067987 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.067970 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.068156 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.068137 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cm46\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-kube-api-access-6cm46\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.068280 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.068262 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.068411 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.068394 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89a6f30-fbab-4350-b331-aa0f2a19b656-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.068944 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.068913 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.069042 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.062870 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.069270 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.063101 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.069491 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.062618 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.069862 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.069838 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89a6f30-fbab-4350-b331-aa0f2a19b656-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.077553 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.077528 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.106085 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.106049 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.115317 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.115289 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.116473 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.116446 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cm46\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-kube-api-access-6cm46\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.268604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.267871 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:31.667221 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.667130 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv"] Oct 29 22:56:31.685027 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.684984 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:31.685152 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.685079 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:31.685152 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.685130 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:31.883606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.883142 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:56:31.883606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:31.883220 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:56:32.200601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:32.200467 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" event={"ID":"e89a6f30-fbab-4350-b331-aa0f2a19b656","Type":"ContainerStarted","Data":"bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba"} Oct 29 22:56:32.200601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:32.200536 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" event={"ID":"e89a6f30-fbab-4350-b331-aa0f2a19b656","Type":"ContainerStarted","Data":"fd0f9644435c87ca7ad642f034c90196f76276cd838fe18fa31b1a03152c3de5"} Oct 29 22:56:32.250605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:32.248673 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podStartSLOduration=2.248638374 podStartE2EDuration="2.248638374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:32.246090866 +0000 UTC m=+789.362010476" watchObservedRunningTime="2026-10-29 22:56:32.248638374 +0000 UTC m=+789.364557982" Oct 29 22:56:32.271201 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:32.271166 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:32.271536 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:32.271515 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:56:32.271724 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:32.271698 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:56:33.274607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.274285 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:56:33.274607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.274359 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:56:33.396238 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.396205 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g"] Oct 29 22:56:33.400956 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.400927 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:33.401116 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.400967 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.401396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.401369 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:33.402246 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.402226 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 22:56:33.424053 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.424020 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n7rn\" (UniqueName: \"kubernetes.io/projected/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-kube-api-access-5n7rn\") pod \"maas-api-revoke-sub-407efa54-c6s9g\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.424290 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.424267 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407efa54-c6s9g\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.424375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.424324 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407efa54-c6s9g\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.425210 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.425186 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:56:33.425330 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.425315 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:56:33.428853 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.428830 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g"] Oct 29 22:56:33.525305 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.525203 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407efa54-c6s9g\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.525305 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.525264 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407efa54-c6s9g\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.525541 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.525381 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n7rn\" (UniqueName: \"kubernetes.io/projected/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-kube-api-access-5n7rn\") pod \"maas-api-revoke-sub-407efa54-c6s9g\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.526309 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.526279 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407efa54-c6s9g\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.527717 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.527694 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407efa54-c6s9g\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.573596 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.573421 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n7rn\" (UniqueName: \"kubernetes.io/projected/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-kube-api-access-5n7rn\") pod \"maas-api-revoke-sub-407efa54-c6s9g\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:33.715155 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:33.715125 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:34.065417 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.065294 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g"] Oct 29 22:56:34.072434 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:34.072376 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a7f082_36b4_40b4_9d1a_f87f38eb2735.slice/crio-2aa4a5660a1a5a9dcb08efa14cbf9559437bbf20ef2d2016417fcb6e2b037446 WatchSource:0}: Error finding container 2aa4a5660a1a5a9dcb08efa14cbf9559437bbf20ef2d2016417fcb6e2b037446: Status 404 returned error can't find the container with id 2aa4a5660a1a5a9dcb08efa14cbf9559437bbf20ef2d2016417fcb6e2b037446 Oct 29 22:56:34.085059 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.085026 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5"] Oct 29 22:56:34.092058 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.091331 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.110361 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.110332 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5"] Oct 29 22:56:34.169471 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.169376 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e31446-g47l5\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.169965 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.169939 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bb2g\" (UniqueName: \"kubernetes.io/projected/bac4e702-7212-49e6-a211-b6b0ac285e01-kube-api-access-6bb2g\") pod \"maas-api-revoke-sub-f0e31446-g47l5\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.170177 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.170160 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e31446-g47l5\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.217087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.216962 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" event={"ID":"c5a7f082-36b4-40b4-9d1a-f87f38eb2735","Type":"ContainerStarted","Data":"2aa4a5660a1a5a9dcb08efa14cbf9559437bbf20ef2d2016417fcb6e2b037446"} Oct 29 22:56:34.269456 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.269417 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:56:34.269760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.269728 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:56:34.272100 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.272064 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bb2g\" (UniqueName: \"kubernetes.io/projected/bac4e702-7212-49e6-a211-b6b0ac285e01-kube-api-access-6bb2g\") pod \"maas-api-revoke-sub-f0e31446-g47l5\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.272224 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.272131 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e31446-g47l5\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.272295 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.272218 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e31446-g47l5\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.273324 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.273264 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e31446-g47l5\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.275513 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.275492 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e31446-g47l5\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.295773 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.295751 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bb2g\" (UniqueName: \"kubernetes.io/projected/bac4e702-7212-49e6-a211-b6b0ac285e01-kube-api-access-6bb2g\") pod \"maas-api-revoke-sub-f0e31446-g47l5\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.428683 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.428637 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:34.834668 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:34.831106 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5"] Oct 29 22:56:35.241631 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:35.230754 5736 generic.go:356] "Generic (PLEG): container finished" podID="c5a7f082-36b4-40b4-9d1a-f87f38eb2735" containerID="5f09dc578592d0712fb622d5c63775dabd6dc0d2ba41ec9fe75c3ca3ad99798d" exitCode=0 Oct 29 22:56:35.264618 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:35.241848 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" event={"ID":"c5a7f082-36b4-40b4-9d1a-f87f38eb2735","Type":"ContainerDied","Data":"5f09dc578592d0712fb622d5c63775dabd6dc0d2ba41ec9fe75c3ca3ad99798d"} Oct 29 22:56:35.269227 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:35.269191 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:56:35.269414 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:35.269247 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:56:35.270715 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:35.270683 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" event={"ID":"bac4e702-7212-49e6-a211-b6b0ac285e01","Type":"ContainerStarted","Data":"961ad5b8e16d2769a8ca25edf2e458dd54ad300181f5ae7f69e1215c435ba9ba"} Oct 29 22:56:36.269397 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.269361 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:56:36.269794 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.269420 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:56:36.278514 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.278485 5736 generic.go:356] "Generic (PLEG): container finished" podID="bac4e702-7212-49e6-a211-b6b0ac285e01" containerID="35fa16cf3aa96e5641e34ed0006172c332052e307b8cd127d41abf5f03973106" exitCode=0 Oct 29 22:56:36.278692 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.278549 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" event={"ID":"bac4e702-7212-49e6-a211-b6b0ac285e01","Type":"ContainerDied","Data":"35fa16cf3aa96e5641e34ed0006172c332052e307b8cd127d41abf5f03973106"} Oct 29 22:56:36.488700 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.488672 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:36.602359 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.602271 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-kube-api-access-5n7rn\" (UniqueName: \"kubernetes.io/projected/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-kube-api-access-5n7rn\") pod \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " Oct 29 22:56:36.602527 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.602395 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-openshift-ca\") pod \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " Oct 29 22:56:36.602527 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.602456 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-xks-ca\") pod \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\" (UID: \"c5a7f082-36b4-40b4-9d1a-f87f38eb2735\") " Oct 29 22:56:36.602854 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.602829 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-openshift-ca" pod "c5a7f082-36b4-40b4-9d1a-f87f38eb2735" (UID: "c5a7f082-36b4-40b4-9d1a-f87f38eb2735"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:36.602996 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.602978 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:36.604399 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.604377 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-kube-api-access-5n7rn" pod "c5a7f082-36b4-40b4-9d1a-f87f38eb2735" (UID: "c5a7f082-36b4-40b4-9d1a-f87f38eb2735"). InnerVolumeSpecName "kube-api-access-5n7rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:36.604475 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.604423 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-xks-ca" pod "c5a7f082-36b4-40b4-9d1a-f87f38eb2735" (UID: "c5a7f082-36b4-40b4-9d1a-f87f38eb2735"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:36.704526 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.704491 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:36.704526 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:36.704522 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n7rn\" (UniqueName: \"kubernetes.io/projected/c5a7f082-36b4-40b4-9d1a-f87f38eb2735-kube-api-access-5n7rn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:37.269014 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.268930 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:56:37.269014 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.268986 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:56:37.285908 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.285876 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" Oct 29 22:56:37.286268 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.285904 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g" event={"ID":"c5a7f082-36b4-40b4-9d1a-f87f38eb2735","Type":"ContainerDied","Data":"2aa4a5660a1a5a9dcb08efa14cbf9559437bbf20ef2d2016417fcb6e2b037446"} Oct 29 22:56:37.286268 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.285956 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aa4a5660a1a5a9dcb08efa14cbf9559437bbf20ef2d2016417fcb6e2b037446" Oct 29 22:56:37.459777 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.459746 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q"] Oct 29 22:56:37.460345 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.460325 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c5a7f082-36b4-40b4-9d1a-f87f38eb2735" containerName="revoke-keys" Oct 29 22:56:37.460448 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.460346 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a7f082-36b4-40b4-9d1a-f87f38eb2735" containerName="revoke-keys" Oct 29 22:56:37.460508 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.460484 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a7f082-36b4-40b4-9d1a-f87f38eb2735" containerName="revoke-keys" Oct 29 22:56:37.462613 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.462531 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:37.463751 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.463729 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.483727 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.483697 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q"] Oct 29 22:56:37.514927 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.514894 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac4e702-7212-49e6-a211-b6b0ac285e01-kube-api-access-6bb2g\" (UniqueName: \"kubernetes.io/projected/bac4e702-7212-49e6-a211-b6b0ac285e01-kube-api-access-6bb2g\") pod \"bac4e702-7212-49e6-a211-b6b0ac285e01\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " Oct 29 22:56:37.515078 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.514938 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-openshift-ca\") pod \"bac4e702-7212-49e6-a211-b6b0ac285e01\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " Oct 29 22:56:37.515158 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.515108 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-xks-ca\") pod \"bac4e702-7212-49e6-a211-b6b0ac285e01\" (UID: \"bac4e702-7212-49e6-a211-b6b0ac285e01\") " Oct 29 22:56:37.515368 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.515345 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-openshift-ca" pod "bac4e702-7212-49e6-a211-b6b0ac285e01" (UID: "bac4e702-7212-49e6-a211-b6b0ac285e01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:37.515448 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.515408 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kggbb\" (UniqueName: \"kubernetes.io/projected/3f6006b7-2f00-4764-8b9f-20f8e5878211-kube-api-access-kggbb\") pod \"maas-api-revoke-sub-1ec2f699-jt28q\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.515550 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.515525 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ec2f699-jt28q\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.515637 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.515613 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ec2f699-jt28q\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.515795 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.515781 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:37.517833 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.517806 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-xks-ca" pod "bac4e702-7212-49e6-a211-b6b0ac285e01" (UID: "bac4e702-7212-49e6-a211-b6b0ac285e01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:37.519061 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.518999 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac4e702-7212-49e6-a211-b6b0ac285e01-kube-api-access-6bb2g" pod "bac4e702-7212-49e6-a211-b6b0ac285e01" (UID: "bac4e702-7212-49e6-a211-b6b0ac285e01"). InnerVolumeSpecName "kube-api-access-6bb2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:37.617251 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.617214 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kggbb\" (UniqueName: \"kubernetes.io/projected/3f6006b7-2f00-4764-8b9f-20f8e5878211-kube-api-access-kggbb\") pod \"maas-api-revoke-sub-1ec2f699-jt28q\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.617398 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.617372 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ec2f699-jt28q\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.617462 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.617447 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ec2f699-jt28q\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.617611 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.617573 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac4e702-7212-49e6-a211-b6b0ac285e01-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:37.617611 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.617611 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bb2g\" (UniqueName: \"kubernetes.io/projected/bac4e702-7212-49e6-a211-b6b0ac285e01-kube-api-access-6bb2g\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:37.618175 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.618146 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ec2f699-jt28q\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.619674 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.619656 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ec2f699-jt28q\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.658147 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.658106 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kggbb\" (UniqueName: \"kubernetes.io/projected/3f6006b7-2f00-4764-8b9f-20f8e5878211-kube-api-access-kggbb\") pod \"maas-api-revoke-sub-1ec2f699-jt28q\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:37.776430 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:37.776349 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:38.112066 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:38.111980 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q"] Oct 29 22:56:38.116924 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:38.116891 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6006b7_2f00_4764_8b9f_20f8e5878211.slice/crio-bcf5defc5c8ae00ddf2d5039cb398235535e16969f5165a1e57d8e5f740833c8 WatchSource:0}: Error finding container bcf5defc5c8ae00ddf2d5039cb398235535e16969f5165a1e57d8e5f740833c8: Status 404 returned error can't find the container with id bcf5defc5c8ae00ddf2d5039cb398235535e16969f5165a1e57d8e5f740833c8 Oct 29 22:56:38.268862 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:38.268828 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:56:38.269050 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:38.268885 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:56:38.294696 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:38.294657 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" event={"ID":"3f6006b7-2f00-4764-8b9f-20f8e5878211","Type":"ContainerStarted","Data":"bcf5defc5c8ae00ddf2d5039cb398235535e16969f5165a1e57d8e5f740833c8"} Oct 29 22:56:38.296312 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:38.296289 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" Oct 29 22:56:38.296451 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:38.296343 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5" event={"ID":"bac4e702-7212-49e6-a211-b6b0ac285e01","Type":"ContainerDied","Data":"961ad5b8e16d2769a8ca25edf2e458dd54ad300181f5ae7f69e1215c435ba9ba"} Oct 29 22:56:38.296451 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:38.296384 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="961ad5b8e16d2769a8ca25edf2e458dd54ad300181f5ae7f69e1215c435ba9ba" Oct 29 22:56:39.273016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:39.272941 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:39.273330 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:39.273312 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:39.274106 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:39.274085 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 22:56:39.303731 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:39.303699 5736 generic.go:356] "Generic (PLEG): container finished" podID="3f6006b7-2f00-4764-8b9f-20f8e5878211" containerID="3a273d22ff8c6c68a91b67aca75584256633a1837813305b45a69c6b5aea0156" exitCode=0 Oct 29 22:56:39.304176 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:39.303779 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" event={"ID":"3f6006b7-2f00-4764-8b9f-20f8e5878211","Type":"ContainerDied","Data":"3a273d22ff8c6c68a91b67aca75584256633a1837813305b45a69c6b5aea0156"} Oct 29 22:56:40.465963 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.465936 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:40.554735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.554701 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-xks-ca\") pod \"3f6006b7-2f00-4764-8b9f-20f8e5878211\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " Oct 29 22:56:40.554924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.554776 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6006b7-2f00-4764-8b9f-20f8e5878211-kube-api-access-kggbb\" (UniqueName: \"kubernetes.io/projected/3f6006b7-2f00-4764-8b9f-20f8e5878211-kube-api-access-kggbb\") pod \"3f6006b7-2f00-4764-8b9f-20f8e5878211\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " Oct 29 22:56:40.554985 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.554932 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-openshift-ca\") pod \"3f6006b7-2f00-4764-8b9f-20f8e5878211\" (UID: \"3f6006b7-2f00-4764-8b9f-20f8e5878211\") " Oct 29 22:56:40.555321 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.555294 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-openshift-ca" pod "3f6006b7-2f00-4764-8b9f-20f8e5878211" (UID: "3f6006b7-2f00-4764-8b9f-20f8e5878211"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:40.555537 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.555517 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:40.556830 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.556791 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6006b7-2f00-4764-8b9f-20f8e5878211-kube-api-access-kggbb" pod "3f6006b7-2f00-4764-8b9f-20f8e5878211" (UID: "3f6006b7-2f00-4764-8b9f-20f8e5878211"). InnerVolumeSpecName "kube-api-access-kggbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:40.556989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.556967 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-xks-ca" pod "3f6006b7-2f00-4764-8b9f-20f8e5878211" (UID: "3f6006b7-2f00-4764-8b9f-20f8e5878211"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:40.656897 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.656861 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6006b7-2f00-4764-8b9f-20f8e5878211-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:40.656897 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:40.656898 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kggbb\" (UniqueName: \"kubernetes.io/projected/3f6006b7-2f00-4764-8b9f-20f8e5878211-kube-api-access-kggbb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:41.318560 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:41.318472 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" Oct 29 22:56:41.318560 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:41.318507 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q" event={"ID":"3f6006b7-2f00-4764-8b9f-20f8e5878211","Type":"ContainerDied","Data":"bcf5defc5c8ae00ddf2d5039cb398235535e16969f5165a1e57d8e5f740833c8"} Oct 29 22:56:41.318560 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:41.318557 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf5defc5c8ae00ddf2d5039cb398235535e16969f5165a1e57d8e5f740833c8" Oct 29 22:56:44.499030 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.498991 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc"] Oct 29 22:56:44.499777 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.499760 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bac4e702-7212-49e6-a211-b6b0ac285e01" containerName="revoke-keys" Oct 29 22:56:44.499847 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.499779 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac4e702-7212-49e6-a211-b6b0ac285e01" containerName="revoke-keys" Oct 29 22:56:44.499847 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.499826 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3f6006b7-2f00-4764-8b9f-20f8e5878211" containerName="revoke-keys" Oct 29 22:56:44.499847 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.499835 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6006b7-2f00-4764-8b9f-20f8e5878211" containerName="revoke-keys" Oct 29 22:56:44.499982 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.499969 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6006b7-2f00-4764-8b9f-20f8e5878211" containerName="revoke-keys" Oct 29 22:56:44.500023 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.499992 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac4e702-7212-49e6-a211-b6b0ac285e01" containerName="revoke-keys" Oct 29 22:56:44.503785 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.503762 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.527180 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.527147 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:56:44.531943 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.531903 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc"] Oct 29 22:56:44.532114 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.532097 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:56:44.603860 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.603827 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksf5p\" (UniqueName: \"kubernetes.io/projected/7feaea65-696f-479e-a204-035b2b100318-kube-api-access-ksf5p\") pod \"maas-api-revoke-sub-d19a9815-qqdbc\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.604110 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.604085 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaea65-696f-479e-a204-035b2b100318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d19a9815-qqdbc\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.604186 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.604134 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaea65-696f-479e-a204-035b2b100318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d19a9815-qqdbc\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.705373 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.705328 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaea65-696f-479e-a204-035b2b100318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d19a9815-qqdbc\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.705678 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.705393 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaea65-696f-479e-a204-035b2b100318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d19a9815-qqdbc\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.705836 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.705796 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksf5p\" (UniqueName: \"kubernetes.io/projected/7feaea65-696f-479e-a204-035b2b100318-kube-api-access-ksf5p\") pod \"maas-api-revoke-sub-d19a9815-qqdbc\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.706638 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.706609 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaea65-696f-479e-a204-035b2b100318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d19a9815-qqdbc\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.709341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.708132 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaea65-696f-479e-a204-035b2b100318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d19a9815-qqdbc\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.740183 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.740145 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksf5p\" (UniqueName: \"kubernetes.io/projected/7feaea65-696f-479e-a204-035b2b100318-kube-api-access-ksf5p\") pod \"maas-api-revoke-sub-d19a9815-qqdbc\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:44.817294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:44.817190 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:45.132018 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:45.131889 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc"] Oct 29 22:56:45.345280 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:45.345244 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" event={"ID":"7feaea65-696f-479e-a204-035b2b100318","Type":"ContainerStarted","Data":"7070d03ebffb004d8cc269389dd92ae16e1b15d65f728036148f16a30f07fecc"} Oct 29 22:56:46.353506 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:46.353468 5736 generic.go:356] "Generic (PLEG): container finished" podID="7feaea65-696f-479e-a204-035b2b100318" containerID="87ca97b3d649405b391ba8534040ecb6891ca7aa006b3fd8fb532d5cd93ef53a" exitCode=0 Oct 29 22:56:46.353986 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:46.353555 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" event={"ID":"7feaea65-696f-479e-a204-035b2b100318","Type":"ContainerDied","Data":"87ca97b3d649405b391ba8534040ecb6891ca7aa006b3fd8fb532d5cd93ef53a"} Oct 29 22:56:47.454271 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.454237 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz"] Oct 29 22:56:47.458945 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.458918 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.477483 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.477451 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz"] Oct 29 22:56:47.515959 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.515933 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:47.544407 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.544377 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-129f77ae-gkjlz\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.544574 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.544490 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-129f77ae-gkjlz\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.544574 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.544528 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgfh8\" (UniqueName: \"kubernetes.io/projected/44da8cd4-821a-4139-8c1d-79e4a41179ed-kube-api-access-kgfh8\") pod \"maas-api-revoke-sub-129f77ae-gkjlz\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.645359 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.645265 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7feaea65-696f-479e-a204-035b2b100318-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaea65-696f-479e-a204-035b2b100318-maas-api-xks-ca\") pod \"7feaea65-696f-479e-a204-035b2b100318\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " Oct 29 22:56:47.645518 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.645387 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7feaea65-696f-479e-a204-035b2b100318-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaea65-696f-479e-a204-035b2b100318-maas-api-openshift-ca\") pod \"7feaea65-696f-479e-a204-035b2b100318\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " Oct 29 22:56:47.645518 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.645512 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7feaea65-696f-479e-a204-035b2b100318-kube-api-access-ksf5p\" (UniqueName: \"kubernetes.io/projected/7feaea65-696f-479e-a204-035b2b100318-kube-api-access-ksf5p\") pod \"7feaea65-696f-479e-a204-035b2b100318\" (UID: \"7feaea65-696f-479e-a204-035b2b100318\") " Oct 29 22:56:47.645780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.645750 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7feaea65-696f-479e-a204-035b2b100318-maas-api-openshift-ca" pod "7feaea65-696f-479e-a204-035b2b100318" (UID: "7feaea65-696f-479e-a204-035b2b100318"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:47.645780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.645771 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-129f77ae-gkjlz\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.645915 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.645855 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-129f77ae-gkjlz\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.645915 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.645878 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgfh8\" (UniqueName: \"kubernetes.io/projected/44da8cd4-821a-4139-8c1d-79e4a41179ed-kube-api-access-kgfh8\") pod \"maas-api-revoke-sub-129f77ae-gkjlz\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.646019 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.645949 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaea65-696f-479e-a204-035b2b100318-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:47.646520 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.646499 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-129f77ae-gkjlz\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.647676 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.647648 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7feaea65-696f-479e-a204-035b2b100318-maas-api-xks-ca" pod "7feaea65-696f-479e-a204-035b2b100318" (UID: "7feaea65-696f-479e-a204-035b2b100318"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:47.647761 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.647690 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7feaea65-696f-479e-a204-035b2b100318-kube-api-access-ksf5p" pod "7feaea65-696f-479e-a204-035b2b100318" (UID: "7feaea65-696f-479e-a204-035b2b100318"). InnerVolumeSpecName "kube-api-access-ksf5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:47.648090 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.648066 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-129f77ae-gkjlz\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.667410 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.667382 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgfh8\" (UniqueName: \"kubernetes.io/projected/44da8cd4-821a-4139-8c1d-79e4a41179ed-kube-api-access-kgfh8\") pod \"maas-api-revoke-sub-129f77ae-gkjlz\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:47.746991 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.746959 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksf5p\" (UniqueName: \"kubernetes.io/projected/7feaea65-696f-479e-a204-035b2b100318-kube-api-access-ksf5p\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:47.746991 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.746988 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaea65-696f-479e-a204-035b2b100318-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:47.782242 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:47.782208 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:48.084497 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:48.084431 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz"] Oct 29 22:56:48.087416 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:48.087387 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44da8cd4_821a_4139_8c1d_79e4a41179ed.slice/crio-228bf3f728109df1cc4f4a85cc14a993870bfef550cb4d160c6ed4ebb27efd73 WatchSource:0}: Error finding container 228bf3f728109df1cc4f4a85cc14a993870bfef550cb4d160c6ed4ebb27efd73: Status 404 returned error can't find the container with id 228bf3f728109df1cc4f4a85cc14a993870bfef550cb4d160c6ed4ebb27efd73 Oct 29 22:56:48.366552 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:48.366525 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" Oct 29 22:56:48.366552 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:48.366544 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc" event={"ID":"7feaea65-696f-479e-a204-035b2b100318","Type":"ContainerDied","Data":"7070d03ebffb004d8cc269389dd92ae16e1b15d65f728036148f16a30f07fecc"} Oct 29 22:56:48.366775 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:48.366611 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7070d03ebffb004d8cc269389dd92ae16e1b15d65f728036148f16a30f07fecc" Oct 29 22:56:48.368342 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:48.368307 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" event={"ID":"44da8cd4-821a-4139-8c1d-79e4a41179ed","Type":"ContainerStarted","Data":"06f67f8e2f951e027a6be39e3aa0a1a5ccec098429ac8b87ee1a7e8059bec040"} Oct 29 22:56:48.368457 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:48.368353 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" event={"ID":"44da8cd4-821a-4139-8c1d-79e4a41179ed","Type":"ContainerStarted","Data":"228bf3f728109df1cc4f4a85cc14a993870bfef550cb4d160c6ed4ebb27efd73"} Oct 29 22:56:48.405597 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:48.405519 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" podStartSLOduration=1.405504553 podStartE2EDuration="1.405504553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:48.403693823 +0000 UTC m=+805.519613432" watchObservedRunningTime="2026-10-29 22:56:48.405504553 +0000 UTC m=+805.521424150" Oct 29 22:56:49.377729 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:49.377693 5736 generic.go:356] "Generic (PLEG): container finished" podID="44da8cd4-821a-4139-8c1d-79e4a41179ed" containerID="06f67f8e2f951e027a6be39e3aa0a1a5ccec098429ac8b87ee1a7e8059bec040" exitCode=0 Oct 29 22:56:49.378134 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:49.377781 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" event={"ID":"44da8cd4-821a-4139-8c1d-79e4a41179ed","Type":"ContainerDied","Data":"06f67f8e2f951e027a6be39e3aa0a1a5ccec098429ac8b87ee1a7e8059bec040"} Oct 29 22:56:50.530841 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.530814 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:50.682885 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.682805 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-xks-ca\") pod \"44da8cd4-821a-4139-8c1d-79e4a41179ed\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " Oct 29 22:56:50.682885 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.682879 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-openshift-ca\") pod \"44da8cd4-821a-4139-8c1d-79e4a41179ed\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " Oct 29 22:56:50.683129 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.682920 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44da8cd4-821a-4139-8c1d-79e4a41179ed-kube-api-access-kgfh8\" (UniqueName: \"kubernetes.io/projected/44da8cd4-821a-4139-8c1d-79e4a41179ed-kube-api-access-kgfh8\") pod \"44da8cd4-821a-4139-8c1d-79e4a41179ed\" (UID: \"44da8cd4-821a-4139-8c1d-79e4a41179ed\") " Oct 29 22:56:50.683239 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.683216 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-openshift-ca" pod "44da8cd4-821a-4139-8c1d-79e4a41179ed" (UID: "44da8cd4-821a-4139-8c1d-79e4a41179ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:50.683342 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.683327 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:50.684957 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.684934 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-xks-ca" pod "44da8cd4-821a-4139-8c1d-79e4a41179ed" (UID: "44da8cd4-821a-4139-8c1d-79e4a41179ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:50.685037 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.684958 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44da8cd4-821a-4139-8c1d-79e4a41179ed-kube-api-access-kgfh8" pod "44da8cd4-821a-4139-8c1d-79e4a41179ed" (UID: "44da8cd4-821a-4139-8c1d-79e4a41179ed"). InnerVolumeSpecName "kube-api-access-kgfh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:50.785724 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.785690 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgfh8\" (UniqueName: \"kubernetes.io/projected/44da8cd4-821a-4139-8c1d-79e4a41179ed-kube-api-access-kgfh8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:50.785724 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:50.785720 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44da8cd4-821a-4139-8c1d-79e4a41179ed-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:51.393241 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:51.393159 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" Oct 29 22:56:51.393395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:51.393157 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz" event={"ID":"44da8cd4-821a-4139-8c1d-79e4a41179ed","Type":"ContainerDied","Data":"228bf3f728109df1cc4f4a85cc14a993870bfef550cb4d160c6ed4ebb27efd73"} Oct 29 22:56:51.393395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:51.393301 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228bf3f728109df1cc4f4a85cc14a993870bfef550cb4d160c6ed4ebb27efd73" Oct 29 22:56:52.203187 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.203150 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w"] Oct 29 22:56:52.203683 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.203668 5736 cpu_manager.go:414] "containerMap: removing container" podUID="44da8cd4-821a-4139-8c1d-79e4a41179ed" containerName="revoke-keys" Oct 29 22:56:52.203726 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.203683 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44da8cd4-821a-4139-8c1d-79e4a41179ed" containerName="revoke-keys" Oct 29 22:56:52.203726 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.203707 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7feaea65-696f-479e-a204-035b2b100318" containerName="revoke-keys" Oct 29 22:56:52.203726 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.203713 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7feaea65-696f-479e-a204-035b2b100318" containerName="revoke-keys" Oct 29 22:56:52.203820 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.203784 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="44da8cd4-821a-4139-8c1d-79e4a41179ed" containerName="revoke-keys" Oct 29 22:56:52.203820 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.203796 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7feaea65-696f-479e-a204-035b2b100318" containerName="revoke-keys" Oct 29 22:56:52.208223 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.208194 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.233365 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.233337 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:56:52.233547 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.233401 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:56:52.237964 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.237931 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w"] Oct 29 22:56:52.299797 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.299764 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs"] Oct 29 22:56:52.303818 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.303782 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19297c31-04a0-4acc-9506-c5149687f819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-181cb988-vtj7w\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.303951 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.303862 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19297c31-04a0-4acc-9506-c5149687f819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-181cb988-vtj7w\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.304022 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.303998 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntsl7\" (UniqueName: \"kubernetes.io/projected/19297c31-04a0-4acc-9506-c5149687f819-kube-api-access-ntsl7\") pod \"maas-api-revoke-sub-181cb988-vtj7w\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.304088 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.304074 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.324920 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.324885 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs"] Oct 29 22:56:52.405053 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.405019 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19297c31-04a0-4acc-9506-c5149687f819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-181cb988-vtj7w\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.405253 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.405070 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df0510c-mpjfs\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.405253 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.405141 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgsmg\" (UniqueName: \"kubernetes.io/projected/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-kube-api-access-lgsmg\") pod \"maas-api-revoke-sub-7df0510c-mpjfs\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.405400 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.405263 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19297c31-04a0-4acc-9506-c5149687f819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-181cb988-vtj7w\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.405400 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.405311 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntsl7\" (UniqueName: \"kubernetes.io/projected/19297c31-04a0-4acc-9506-c5149687f819-kube-api-access-ntsl7\") pod \"maas-api-revoke-sub-181cb988-vtj7w\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.405496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.405475 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df0510c-mpjfs\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.406381 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.406343 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19297c31-04a0-4acc-9506-c5149687f819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-181cb988-vtj7w\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.407763 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.407739 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19297c31-04a0-4acc-9506-c5149687f819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-181cb988-vtj7w\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.427058 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.427021 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntsl7\" (UniqueName: \"kubernetes.io/projected/19297c31-04a0-4acc-9506-c5149687f819-kube-api-access-ntsl7\") pod \"maas-api-revoke-sub-181cb988-vtj7w\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.507133 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.507095 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df0510c-mpjfs\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.507336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.507197 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df0510c-mpjfs\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.507336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.507249 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgsmg\" (UniqueName: \"kubernetes.io/projected/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-kube-api-access-lgsmg\") pod \"maas-api-revoke-sub-7df0510c-mpjfs\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.508111 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.508083 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df0510c-mpjfs\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.509939 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.509911 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df0510c-mpjfs\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.523668 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.523646 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:52.528421 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.528400 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgsmg\" (UniqueName: \"kubernetes.io/projected/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-kube-api-access-lgsmg\") pod \"maas-api-revoke-sub-7df0510c-mpjfs\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.617242 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.617212 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:52.831260 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.831230 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w"] Oct 29 22:56:52.834020 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:52.833991 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19297c31_04a0_4acc_9506_c5149687f819.slice/crio-fb1af69c873ec8e5775849067c36a12155dc460b5f3bc8b2083f2ed26c1deba4 WatchSource:0}: Error finding container fb1af69c873ec8e5775849067c36a12155dc460b5f3bc8b2083f2ed26c1deba4: Status 404 returned error can't find the container with id fb1af69c873ec8e5775849067c36a12155dc460b5f3bc8b2083f2ed26c1deba4 Oct 29 22:56:52.925402 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:52.925370 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs"] Oct 29 22:56:52.928515 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:52.928492 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60142a70_c4d6_4efd_9ced_6fb6e72fdc23.slice/crio-3e6e827fbde5bba6091c3200421f8affcef9828c2f767dad9d967a88b43b1fac WatchSource:0}: Error finding container 3e6e827fbde5bba6091c3200421f8affcef9828c2f767dad9d967a88b43b1fac: Status 404 returned error can't find the container with id 3e6e827fbde5bba6091c3200421f8affcef9828c2f767dad9d967a88b43b1fac Oct 29 22:56:53.407591 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:53.407475 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" event={"ID":"60142a70-c4d6-4efd-9ced-6fb6e72fdc23","Type":"ContainerStarted","Data":"9e6664f49a12d352470781355edd6f700ae6f9a7ba71d1a0d703be62b7c03453"} Oct 29 22:56:53.407591 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:53.407541 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" event={"ID":"60142a70-c4d6-4efd-9ced-6fb6e72fdc23","Type":"ContainerStarted","Data":"3e6e827fbde5bba6091c3200421f8affcef9828c2f767dad9d967a88b43b1fac"} Oct 29 22:56:53.409244 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:53.409211 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" event={"ID":"19297c31-04a0-4acc-9506-c5149687f819","Type":"ContainerStarted","Data":"6981ad784b9d21288901fb6d4844402eafb5cbe2cb052bcd2cd826f88c973e29"} Oct 29 22:56:53.409374 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:53.409270 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" event={"ID":"19297c31-04a0-4acc-9506-c5149687f819","Type":"ContainerStarted","Data":"fb1af69c873ec8e5775849067c36a12155dc460b5f3bc8b2083f2ed26c1deba4"} Oct 29 22:56:53.447048 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:53.446993 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" podStartSLOduration=1.446981571 podStartE2EDuration="1.446981571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:53.446293732 +0000 UTC m=+810.562213340" watchObservedRunningTime="2026-10-29 22:56:53.446981571 +0000 UTC m=+810.562901176" Oct 29 22:56:53.486068 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:53.486008 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" podStartSLOduration=1.485992038 podStartE2EDuration="1.485992038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:53.483743468 +0000 UTC m=+810.599663071" watchObservedRunningTime="2026-10-29 22:56:53.485992038 +0000 UTC m=+810.601911645" Oct 29 22:56:54.417447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:54.417413 5736 generic.go:356] "Generic (PLEG): container finished" podID="60142a70-c4d6-4efd-9ced-6fb6e72fdc23" containerID="9e6664f49a12d352470781355edd6f700ae6f9a7ba71d1a0d703be62b7c03453" exitCode=0 Oct 29 22:56:54.417955 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:54.417494 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" event={"ID":"60142a70-c4d6-4efd-9ced-6fb6e72fdc23","Type":"ContainerDied","Data":"9e6664f49a12d352470781355edd6f700ae6f9a7ba71d1a0d703be62b7c03453"} Oct 29 22:56:54.419111 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:54.419090 5736 generic.go:356] "Generic (PLEG): container finished" podID="19297c31-04a0-4acc-9506-c5149687f819" containerID="6981ad784b9d21288901fb6d4844402eafb5cbe2cb052bcd2cd826f88c973e29" exitCode=0 Oct 29 22:56:54.419222 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:54.419167 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" event={"ID":"19297c31-04a0-4acc-9506-c5149687f819","Type":"ContainerDied","Data":"6981ad784b9d21288901fb6d4844402eafb5cbe2cb052bcd2cd826f88c973e29"} Oct 29 22:56:55.574054 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.574025 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:55.623105 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.623069 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:55.745980 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.745947 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19297c31-04a0-4acc-9506-c5149687f819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19297c31-04a0-4acc-9506-c5149687f819-maas-api-xks-ca\") pod \"19297c31-04a0-4acc-9506-c5149687f819\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " Oct 29 22:56:55.746174 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.746156 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-kube-api-access-lgsmg\" (UniqueName: \"kubernetes.io/projected/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-kube-api-access-lgsmg\") pod \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " Oct 29 22:56:55.746236 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.746222 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-openshift-ca\") pod \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " Oct 29 22:56:55.746281 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.746269 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19297c31-04a0-4acc-9506-c5149687f819-kube-api-access-ntsl7\" (UniqueName: \"kubernetes.io/projected/19297c31-04a0-4acc-9506-c5149687f819-kube-api-access-ntsl7\") pod \"19297c31-04a0-4acc-9506-c5149687f819\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " Oct 29 22:56:55.746338 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.746306 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19297c31-04a0-4acc-9506-c5149687f819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19297c31-04a0-4acc-9506-c5149687f819-maas-api-openshift-ca\") pod \"19297c31-04a0-4acc-9506-c5149687f819\" (UID: \"19297c31-04a0-4acc-9506-c5149687f819\") " Oct 29 22:56:55.746392 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.746365 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-xks-ca\") pod \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\" (UID: \"60142a70-c4d6-4efd-9ced-6fb6e72fdc23\") " Oct 29 22:56:55.746623 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.746553 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-openshift-ca" pod "60142a70-c4d6-4efd-9ced-6fb6e72fdc23" (UID: "60142a70-c4d6-4efd-9ced-6fb6e72fdc23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:55.746776 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.746744 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19297c31-04a0-4acc-9506-c5149687f819-maas-api-openshift-ca" pod "19297c31-04a0-4acc-9506-c5149687f819" (UID: "19297c31-04a0-4acc-9506-c5149687f819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:55.747000 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.746981 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:55.747069 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.747005 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19297c31-04a0-4acc-9506-c5149687f819-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:55.748395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.748367 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-kube-api-access-lgsmg" pod "60142a70-c4d6-4efd-9ced-6fb6e72fdc23" (UID: "60142a70-c4d6-4efd-9ced-6fb6e72fdc23"). InnerVolumeSpecName "kube-api-access-lgsmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:55.748493 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.748473 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19297c31-04a0-4acc-9506-c5149687f819-maas-api-xks-ca" pod "19297c31-04a0-4acc-9506-c5149687f819" (UID: "19297c31-04a0-4acc-9506-c5149687f819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:55.748843 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.748816 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-xks-ca" pod "60142a70-c4d6-4efd-9ced-6fb6e72fdc23" (UID: "60142a70-c4d6-4efd-9ced-6fb6e72fdc23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:55.748933 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.748872 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19297c31-04a0-4acc-9506-c5149687f819-kube-api-access-ntsl7" pod "19297c31-04a0-4acc-9506-c5149687f819" (UID: "19297c31-04a0-4acc-9506-c5149687f819"). InnerVolumeSpecName "kube-api-access-ntsl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:55.848621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.848572 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19297c31-04a0-4acc-9506-c5149687f819-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:55.848621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.848618 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgsmg\" (UniqueName: \"kubernetes.io/projected/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-kube-api-access-lgsmg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:55.848808 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.848635 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntsl7\" (UniqueName: \"kubernetes.io/projected/19297c31-04a0-4acc-9506-c5149687f819-kube-api-access-ntsl7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:55.848808 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:55.848648 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60142a70-c4d6-4efd-9ced-6fb6e72fdc23-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:56:56.432654 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:56.432610 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" event={"ID":"60142a70-c4d6-4efd-9ced-6fb6e72fdc23","Type":"ContainerDied","Data":"3e6e827fbde5bba6091c3200421f8affcef9828c2f767dad9d967a88b43b1fac"} Oct 29 22:56:56.432654 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:56.432648 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs" Oct 29 22:56:56.432889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:56.432674 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6e827fbde5bba6091c3200421f8affcef9828c2f767dad9d967a88b43b1fac" Oct 29 22:56:56.434113 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:56.434085 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" event={"ID":"19297c31-04a0-4acc-9506-c5149687f819","Type":"ContainerDied","Data":"fb1af69c873ec8e5775849067c36a12155dc460b5f3bc8b2083f2ed26c1deba4"} Oct 29 22:56:56.434233 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:56.434133 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb1af69c873ec8e5775849067c36a12155dc460b5f3bc8b2083f2ed26c1deba4" Oct 29 22:56:56.434233 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:56.434096 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w" Oct 29 22:56:56.823305 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:56:56.823271 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:56:57.803092 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.803060 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn"] Oct 29 22:56:57.803570 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.803556 5736 cpu_manager.go:414] "containerMap: removing container" podUID="19297c31-04a0-4acc-9506-c5149687f819" containerName="revoke-keys" Oct 29 22:56:57.803646 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.803572 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19297c31-04a0-4acc-9506-c5149687f819" containerName="revoke-keys" Oct 29 22:56:57.803646 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.803613 5736 cpu_manager.go:414] "containerMap: removing container" podUID="60142a70-c4d6-4efd-9ced-6fb6e72fdc23" containerName="revoke-keys" Oct 29 22:56:57.803646 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.803618 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60142a70-c4d6-4efd-9ced-6fb6e72fdc23" containerName="revoke-keys" Oct 29 22:56:57.803736 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.803716 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="60142a70-c4d6-4efd-9ced-6fb6e72fdc23" containerName="revoke-keys" Oct 29 22:56:57.803736 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.803730 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="19297c31-04a0-4acc-9506-c5149687f819" containerName="revoke-keys" Oct 29 22:56:57.808420 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.808400 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:57.823266 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.823242 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:56:57.826487 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.826455 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn"] Oct 29 22:56:57.848618 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.838409 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7"] Oct 29 22:56:57.848618 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.841815 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:56:57.848618 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.843173 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:57.861480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.861453 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7"] Oct 29 22:56:57.972469 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.972432 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d770a527-g7fq7\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:57.972714 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.972531 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e19d84-lt2cn\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:57.972714 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.972567 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbfq\" (UniqueName: \"kubernetes.io/projected/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-kube-api-access-vkbfq\") pod \"maas-api-revoke-sub-f3e19d84-lt2cn\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:57.972714 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.972630 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d770a527-g7fq7\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:57.972714 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.972691 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcbhs\" (UniqueName: \"kubernetes.io/projected/3a35a194-d69e-47f2-ac79-464a4a82b202-kube-api-access-bcbhs\") pod \"maas-api-revoke-sub-d770a527-g7fq7\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:57.972937 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:57.972736 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e19d84-lt2cn\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:58.074087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.073992 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d770a527-g7fq7\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:58.074087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.074060 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e19d84-lt2cn\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:58.074087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.074083 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbfq\" (UniqueName: \"kubernetes.io/projected/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-kube-api-access-vkbfq\") pod \"maas-api-revoke-sub-f3e19d84-lt2cn\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:58.074373 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.074115 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d770a527-g7fq7\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:58.074373 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.074172 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcbhs\" (UniqueName: \"kubernetes.io/projected/3a35a194-d69e-47f2-ac79-464a4a82b202-kube-api-access-bcbhs\") pod \"maas-api-revoke-sub-d770a527-g7fq7\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:58.074373 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.074216 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e19d84-lt2cn\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:58.074911 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.074878 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d770a527-g7fq7\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:58.075012 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.074940 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e19d84-lt2cn\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:58.076611 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.076570 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d770a527-g7fq7\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:58.076787 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.076770 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e19d84-lt2cn\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:58.095910 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.095886 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbfq\" (UniqueName: \"kubernetes.io/projected/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-kube-api-access-vkbfq\") pod \"maas-api-revoke-sub-f3e19d84-lt2cn\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:58.096163 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.096140 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcbhs\" (UniqueName: \"kubernetes.io/projected/3a35a194-d69e-47f2-ac79-464a4a82b202-kube-api-access-bcbhs\") pod \"maas-api-revoke-sub-d770a527-g7fq7\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:58.166318 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.166286 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd"] Oct 29 22:56:58.168078 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.168053 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:56:58.172314 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.172208 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.175358 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.175333 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:56:58.197365 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.197337 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd"] Oct 29 22:56:58.276244 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.276209 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.276413 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.276247 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.276413 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.276301 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.276413 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.276338 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwp7p\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-kube-api-access-wwp7p\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.276413 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.276369 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.276662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.276452 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.276662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.276517 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.276967 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.276940 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.277064 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.277037 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378482 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378396 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378482 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378435 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378482 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378472 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378496 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwp7p\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-kube-api-access-wwp7p\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378516 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378642 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378760 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378711 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378958 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378792 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378958 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378877 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378958 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378896 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.378958 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.378929 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.379141 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.379073 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.379238 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.379222 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.379668 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.379646 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.381012 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.380986 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.381131 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.381076 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.399902 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.399877 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwp7p\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-kube-api-access-wwp7p\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.399902 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.399894 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.490040 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.490007 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:58.524251 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.524165 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn"] Oct 29 22:56:58.527904 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:58.527868 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61e80b5_f3f1_4951_8960_ed0fb44f94a7.slice/crio-d881ab6a5aba543408d19368cbf6aa05473c08d373d2dd61a24df51462da8b00 WatchSource:0}: Error finding container d881ab6a5aba543408d19368cbf6aa05473c08d373d2dd61a24df51462da8b00: Status 404 returned error can't find the container with id d881ab6a5aba543408d19368cbf6aa05473c08d373d2dd61a24df51462da8b00 Oct 29 22:56:58.529075 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.529053 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7"] Oct 29 22:56:58.542056 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:58.542011 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a35a194_d69e_47f2_ac79_464a4a82b202.slice/crio-20b7decb0af87aa034fb5e26de2d94a171aac9f4984186784a89d327db82c7b0 WatchSource:0}: Error finding container 20b7decb0af87aa034fb5e26de2d94a171aac9f4984186784a89d327db82c7b0: Status 404 returned error can't find the container with id 20b7decb0af87aa034fb5e26de2d94a171aac9f4984186784a89d327db82c7b0 Oct 29 22:56:58.799447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.799414 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd"] Oct 29 22:56:58.803271 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:56:58.803244 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d260f4a_dfb6_49d7_b59b_5cdef42f6686.slice/crio-6e6d797bdffb44406e61fde931f2c0ea66223d510db61dce4306a7269269a59a WatchSource:0}: Error finding container 6e6d797bdffb44406e61fde931f2c0ea66223d510db61dce4306a7269269a59a: Status 404 returned error can't find the container with id 6e6d797bdffb44406e61fde931f2c0ea66223d510db61dce4306a7269269a59a Oct 29 22:56:58.805862 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.805832 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:58.805938 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.805891 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:58.805938 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:58.805918 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:56:59.455504 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.455472 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" event={"ID":"8d260f4a-dfb6-49d7-b59b-5cdef42f6686","Type":"ContainerStarted","Data":"a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe"} Oct 29 22:56:59.455914 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.455543 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" event={"ID":"8d260f4a-dfb6-49d7-b59b-5cdef42f6686","Type":"ContainerStarted","Data":"6e6d797bdffb44406e61fde931f2c0ea66223d510db61dce4306a7269269a59a"} Oct 29 22:56:59.457172 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.457140 5736 generic.go:356] "Generic (PLEG): container finished" podID="3a35a194-d69e-47f2-ac79-464a4a82b202" containerID="7b5c63be3eb2fbf050a30f964fc2eceb55a6b180a5c881f4099b32f1560126b3" exitCode=0 Oct 29 22:56:59.457294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.457191 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" event={"ID":"3a35a194-d69e-47f2-ac79-464a4a82b202","Type":"ContainerDied","Data":"7b5c63be3eb2fbf050a30f964fc2eceb55a6b180a5c881f4099b32f1560126b3"} Oct 29 22:56:59.457294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.457248 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" event={"ID":"3a35a194-d69e-47f2-ac79-464a4a82b202","Type":"ContainerStarted","Data":"20b7decb0af87aa034fb5e26de2d94a171aac9f4984186784a89d327db82c7b0"} Oct 29 22:56:59.459066 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.459041 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" event={"ID":"e61e80b5-f3f1-4951-8960-ed0fb44f94a7","Type":"ContainerStarted","Data":"7c50e32a394067028b1fb18426831dd664033bf07b87b594631eafd31a25fcff"} Oct 29 22:56:59.459157 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.459087 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" event={"ID":"e61e80b5-f3f1-4951-8960-ed0fb44f94a7","Type":"ContainerStarted","Data":"d881ab6a5aba543408d19368cbf6aa05473c08d373d2dd61a24df51462da8b00"} Oct 29 22:56:59.490900 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.490863 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:56:59.492358 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.492334 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:56:59.492469 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.492375 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:56:59.507533 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.507482 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podStartSLOduration=1.507466159 podStartE2EDuration="1.507466159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:59.505622298 +0000 UTC m=+816.621541901" watchObservedRunningTime="2026-10-29 22:56:59.507466159 +0000 UTC m=+816.623385768" Oct 29 22:56:59.580919 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:56:59.580868 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" podStartSLOduration=2.58085503 podStartE2EDuration="2.58085503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:59.576646779 +0000 UTC m=+816.692566385" watchObservedRunningTime="2026-10-29 22:56:59.58085503 +0000 UTC m=+816.696774637" Oct 29 22:57:00.467362 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.467329 5736 generic.go:356] "Generic (PLEG): container finished" podID="e61e80b5-f3f1-4951-8960-ed0fb44f94a7" containerID="7c50e32a394067028b1fb18426831dd664033bf07b87b594631eafd31a25fcff" exitCode=0 Oct 29 22:57:00.467819 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.467408 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" event={"ID":"e61e80b5-f3f1-4951-8960-ed0fb44f94a7","Type":"ContainerDied","Data":"7c50e32a394067028b1fb18426831dd664033bf07b87b594631eafd31a25fcff"} Oct 29 22:57:00.490854 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.490812 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:57:00.491076 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.490878 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:57:00.671145 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.671064 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:57:00.706304 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.706274 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-xks-ca\") pod \"3a35a194-d69e-47f2-ac79-464a4a82b202\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " Oct 29 22:57:00.706457 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.706313 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a35a194-d69e-47f2-ac79-464a4a82b202-kube-api-access-bcbhs\" (UniqueName: \"kubernetes.io/projected/3a35a194-d69e-47f2-ac79-464a4a82b202-kube-api-access-bcbhs\") pod \"3a35a194-d69e-47f2-ac79-464a4a82b202\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " Oct 29 22:57:00.706500 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.706470 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-openshift-ca\") pod \"3a35a194-d69e-47f2-ac79-464a4a82b202\" (UID: \"3a35a194-d69e-47f2-ac79-464a4a82b202\") " Oct 29 22:57:00.706911 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.706877 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-openshift-ca" pod "3a35a194-d69e-47f2-ac79-464a4a82b202" (UID: "3a35a194-d69e-47f2-ac79-464a4a82b202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:00.707040 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.706958 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:00.708353 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.708325 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-xks-ca" pod "3a35a194-d69e-47f2-ac79-464a4a82b202" (UID: "3a35a194-d69e-47f2-ac79-464a4a82b202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:00.708497 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.708462 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a35a194-d69e-47f2-ac79-464a4a82b202-kube-api-access-bcbhs" pod "3a35a194-d69e-47f2-ac79-464a4a82b202" (UID: "3a35a194-d69e-47f2-ac79-464a4a82b202"). InnerVolumeSpecName "kube-api-access-bcbhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:00.808690 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.808660 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a35a194-d69e-47f2-ac79-464a4a82b202-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:00.808690 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:00.808688 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcbhs\" (UniqueName: \"kubernetes.io/projected/3a35a194-d69e-47f2-ac79-464a4a82b202-kube-api-access-bcbhs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:01.476070 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.476036 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" Oct 29 22:57:01.476491 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.476073 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7" event={"ID":"3a35a194-d69e-47f2-ac79-464a4a82b202","Type":"ContainerDied","Data":"20b7decb0af87aa034fb5e26de2d94a171aac9f4984186784a89d327db82c7b0"} Oct 29 22:57:01.476491 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.476122 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b7decb0af87aa034fb5e26de2d94a171aac9f4984186784a89d327db82c7b0" Oct 29 22:57:01.491302 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.491258 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:57:01.491478 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.491344 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:57:01.713042 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.713005 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:57:01.822266 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.822180 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-kube-api-access-vkbfq\" (UniqueName: \"kubernetes.io/projected/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-kube-api-access-vkbfq\") pod \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " Oct 29 22:57:01.822266 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.822253 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-xks-ca\") pod \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " Oct 29 22:57:01.822482 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.822461 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-openshift-ca\") pod \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\" (UID: \"e61e80b5-f3f1-4951-8960-ed0fb44f94a7\") " Oct 29 22:57:01.822866 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.822839 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-openshift-ca" pod "e61e80b5-f3f1-4951-8960-ed0fb44f94a7" (UID: "e61e80b5-f3f1-4951-8960-ed0fb44f94a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:01.824864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.824835 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-xks-ca" pod "e61e80b5-f3f1-4951-8960-ed0fb44f94a7" (UID: "e61e80b5-f3f1-4951-8960-ed0fb44f94a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:01.824864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.824855 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-kube-api-access-vkbfq" pod "e61e80b5-f3f1-4951-8960-ed0fb44f94a7" (UID: "e61e80b5-f3f1-4951-8960-ed0fb44f94a7"). InnerVolumeSpecName "kube-api-access-vkbfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:01.924174 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.924137 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:01.924174 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.924167 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbfq\" (UniqueName: \"kubernetes.io/projected/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-kube-api-access-vkbfq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:01.924174 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:01.924180 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61e80b5-f3f1-4951-8960-ed0fb44f94a7-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:02.484785 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:02.484754 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" Oct 29 22:57:02.485183 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:02.484782 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn" event={"ID":"e61e80b5-f3f1-4951-8960-ed0fb44f94a7","Type":"ContainerDied","Data":"d881ab6a5aba543408d19368cbf6aa05473c08d373d2dd61a24df51462da8b00"} Oct 29 22:57:02.485183 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:02.484833 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d881ab6a5aba543408d19368cbf6aa05473c08d373d2dd61a24df51462da8b00" Oct 29 22:57:02.491026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:02.490993 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:57:02.491147 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:02.491053 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:57:03.623605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:03.623438 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:57:03.623605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:03.623501 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:57:04.448709 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.448668 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:04.448991 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.448961 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:04.491061 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.490636 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:57:04.491061 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.490709 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:57:04.595398 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.595355 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 29 22:57:04.751603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.750679 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:04.846607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.844136 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 29 22:57:04.865610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.861517 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:04.899603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.899161 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:04.940592 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.940528 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:04.955702 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.955653 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 22:57:04.998621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.996000 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:04.998954 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:04.998926 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:05.103664 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:05.102660 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:05.497644 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:05.493803 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:57:05.497971 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:05.497921 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:57:05.711339 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:05.711284 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:05.866805 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:05.866656 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:05.868097 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:05.867968 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:05.868978 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:05.868959 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:57:05.894777 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:05.894747 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:57:06.005792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:06.005749 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:06.020600 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:06.015544 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:57:06.070496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:06.070465 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:57:06.490533 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:06.490493 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:57:06.490770 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:06.490550 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:57:07.719376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.719341 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls"] Oct 29 22:57:07.719909 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.719890 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3a35a194-d69e-47f2-ac79-464a4a82b202" containerName="revoke-keys" Oct 29 22:57:07.719909 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.719908 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a35a194-d69e-47f2-ac79-464a4a82b202" containerName="revoke-keys" Oct 29 22:57:07.720035 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.719921 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e61e80b5-f3f1-4951-8960-ed0fb44f94a7" containerName="revoke-keys" Oct 29 22:57:07.720035 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.719929 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61e80b5-f3f1-4951-8960-ed0fb44f94a7" containerName="revoke-keys" Oct 29 22:57:07.720129 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.720046 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a35a194-d69e-47f2-ac79-464a4a82b202" containerName="revoke-keys" Oct 29 22:57:07.720129 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.720056 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61e80b5-f3f1-4951-8960-ed0fb44f94a7" containerName="revoke-keys" Oct 29 22:57:07.724624 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.724605 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:07.743536 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.743505 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls"] Oct 29 22:57:07.749905 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.749881 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:57:07.750051 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.749965 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:57:07.819953 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.819916 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86b21d2-fw6ls\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:07.820206 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.820177 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n7fd\" (UniqueName: \"kubernetes.io/projected/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-kube-api-access-7n7fd\") pod \"maas-api-revoke-sub-f86b21d2-fw6ls\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:07.820309 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.820294 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86b21d2-fw6ls\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:07.921567 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.921513 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n7fd\" (UniqueName: \"kubernetes.io/projected/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-kube-api-access-7n7fd\") pod \"maas-api-revoke-sub-f86b21d2-fw6ls\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:07.922240 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.922211 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86b21d2-fw6ls\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:07.922448 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.922429 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86b21d2-fw6ls\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:07.923533 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.923510 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86b21d2-fw6ls\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:07.926063 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.926044 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86b21d2-fw6ls\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:07.943645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:07.943616 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n7fd\" (UniqueName: \"kubernetes.io/projected/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-kube-api-access-7n7fd\") pod \"maas-api-revoke-sub-f86b21d2-fw6ls\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:08.038475 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.038363 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:08.361231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.361201 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls"] Oct 29 22:57:08.365174 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:08.365141 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55cac58_d1b3_43b4_b3a5_6fbba73240b7.slice/crio-a5d1fea950d49d4099a973c78a611007c2f5a497d583b26ef9e7ab4edb0079f6 WatchSource:0}: Error finding container a5d1fea950d49d4099a973c78a611007c2f5a497d583b26ef9e7ab4edb0079f6: Status 404 returned error can't find the container with id a5d1fea950d49d4099a973c78a611007c2f5a497d583b26ef9e7ab4edb0079f6 Oct 29 22:57:08.531736 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.531701 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:08.531902 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.531760 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:08.561613 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.561547 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:57:08.561893 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.561869 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:57:08.563025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.563004 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 22:57:08.747441 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.747406 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" event={"ID":"e55cac58-d1b3-43b4-b3a5-6fbba73240b7","Type":"ContainerStarted","Data":"09a760bf0641d280f1434a34ce75bd98dc6b9f33c7f1959d8def243e44285ab9"} Oct 29 22:57:08.747891 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.747461 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" event={"ID":"e55cac58-d1b3-43b4-b3a5-6fbba73240b7","Type":"ContainerStarted","Data":"a5d1fea950d49d4099a973c78a611007c2f5a497d583b26ef9e7ab4edb0079f6"} Oct 29 22:57:08.782898 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:08.782855 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" podStartSLOduration=1.7828403160000001 podStartE2EDuration="1.782840316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:08.782457327 +0000 UTC m=+825.898376944" watchObservedRunningTime="2026-10-29 22:57:08.782840316 +0000 UTC m=+825.898759922" Oct 29 22:57:09.765005 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:09.764937 5736 generic.go:356] "Generic (PLEG): container finished" podID="e55cac58-d1b3-43b4-b3a5-6fbba73240b7" containerID="09a760bf0641d280f1434a34ce75bd98dc6b9f33c7f1959d8def243e44285ab9" exitCode=0 Oct 29 22:57:09.765452 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:09.765033 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" event={"ID":"e55cac58-d1b3-43b4-b3a5-6fbba73240b7","Type":"ContainerDied","Data":"09a760bf0641d280f1434a34ce75bd98dc6b9f33c7f1959d8def243e44285ab9"} Oct 29 22:57:10.932774 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:10.932744 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:11.065906 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.065825 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-openshift-ca\") pod \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " Oct 29 22:57:11.065906 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.065860 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-kube-api-access-7n7fd\" (UniqueName: \"kubernetes.io/projected/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-kube-api-access-7n7fd\") pod \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " Oct 29 22:57:11.066092 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.065942 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-xks-ca\") pod \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\" (UID: \"e55cac58-d1b3-43b4-b3a5-6fbba73240b7\") " Oct 29 22:57:11.066293 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.066267 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-openshift-ca" pod "e55cac58-d1b3-43b4-b3a5-6fbba73240b7" (UID: "e55cac58-d1b3-43b4-b3a5-6fbba73240b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:11.066454 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.066436 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:11.068134 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.068110 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-kube-api-access-7n7fd" pod "e55cac58-d1b3-43b4-b3a5-6fbba73240b7" (UID: "e55cac58-d1b3-43b4-b3a5-6fbba73240b7"). InnerVolumeSpecName "kube-api-access-7n7fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:11.068253 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.068185 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-xks-ca" pod "e55cac58-d1b3-43b4-b3a5-6fbba73240b7" (UID: "e55cac58-d1b3-43b4-b3a5-6fbba73240b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:11.168250 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.168214 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n7fd\" (UniqueName: \"kubernetes.io/projected/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-kube-api-access-7n7fd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:11.168250 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.168249 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55cac58-d1b3-43b4-b3a5-6fbba73240b7-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:11.781143 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.781111 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" Oct 29 22:57:11.781342 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.781141 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls" event={"ID":"e55cac58-d1b3-43b4-b3a5-6fbba73240b7","Type":"ContainerDied","Data":"a5d1fea950d49d4099a973c78a611007c2f5a497d583b26ef9e7ab4edb0079f6"} Oct 29 22:57:11.781342 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:11.781190 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d1fea950d49d4099a973c78a611007c2f5a497d583b26ef9e7ab4edb0079f6" Oct 29 22:57:12.851991 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.851959 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j"] Oct 29 22:57:12.852726 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.852704 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e55cac58-d1b3-43b4-b3a5-6fbba73240b7" containerName="revoke-keys" Oct 29 22:57:12.852726 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.852726 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55cac58-d1b3-43b4-b3a5-6fbba73240b7" containerName="revoke-keys" Oct 29 22:57:12.852892 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.852866 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55cac58-d1b3-43b4-b3a5-6fbba73240b7" containerName="revoke-keys" Oct 29 22:57:12.861901 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.861866 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:12.876729 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.876680 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j"] Oct 29 22:57:12.884695 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.884654 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:57:12.884839 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.884718 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:57:12.889459 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.889433 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88cj\" (UniqueName: \"kubernetes.io/projected/43b9725d-7a83-4ccc-bc79-0f855d4fd875-kube-api-access-s88cj\") pod \"maas-api-revoke-sub-aca3e648-n8k9j\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:12.889749 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.889731 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aca3e648-n8k9j\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:12.889846 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.889779 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aca3e648-n8k9j\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:12.990919 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.990880 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aca3e648-n8k9j\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:12.991093 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.990944 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aca3e648-n8k9j\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:12.991093 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.990986 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88cj\" (UniqueName: \"kubernetes.io/projected/43b9725d-7a83-4ccc-bc79-0f855d4fd875-kube-api-access-s88cj\") pod \"maas-api-revoke-sub-aca3e648-n8k9j\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:12.991750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.991724 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aca3e648-n8k9j\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:12.993438 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:12.993413 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aca3e648-n8k9j\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:13.013690 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:13.013663 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88cj\" (UniqueName: \"kubernetes.io/projected/43b9725d-7a83-4ccc-bc79-0f855d4fd875-kube-api-access-s88cj\") pod \"maas-api-revoke-sub-aca3e648-n8k9j\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:13.180489 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:13.180414 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:13.492706 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:13.492655 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j"] Oct 29 22:57:13.495534 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:13.495499 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b9725d_7a83_4ccc_bc79_0f855d4fd875.slice/crio-65d9b3441154a352fae76bd5123aea6bafbf859222ba39f96c31d4dc733dd456 WatchSource:0}: Error finding container 65d9b3441154a352fae76bd5123aea6bafbf859222ba39f96c31d4dc733dd456: Status 404 returned error can't find the container with id 65d9b3441154a352fae76bd5123aea6bafbf859222ba39f96c31d4dc733dd456 Oct 29 22:57:13.795786 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:13.795749 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" event={"ID":"43b9725d-7a83-4ccc-bc79-0f855d4fd875","Type":"ContainerStarted","Data":"bba32dfd1c70cd5e17aaa3b91b7ed28823f860e662ebb7630c40fc4155511cf6"} Oct 29 22:57:13.795931 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:13.795811 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" event={"ID":"43b9725d-7a83-4ccc-bc79-0f855d4fd875","Type":"ContainerStarted","Data":"65d9b3441154a352fae76bd5123aea6bafbf859222ba39f96c31d4dc733dd456"} Oct 29 22:57:13.831318 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:13.831271 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" podStartSLOduration=1.8312587310000001 podStartE2EDuration="1.831258731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:13.830693848 +0000 UTC m=+830.946613453" watchObservedRunningTime="2026-10-29 22:57:13.831258731 +0000 UTC m=+830.947178362" Oct 29 22:57:14.803967 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:14.803933 5736 generic.go:356] "Generic (PLEG): container finished" podID="43b9725d-7a83-4ccc-bc79-0f855d4fd875" containerID="bba32dfd1c70cd5e17aaa3b91b7ed28823f860e662ebb7630c40fc4155511cf6" exitCode=0 Oct 29 22:57:14.804401 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:14.804025 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" event={"ID":"43b9725d-7a83-4ccc-bc79-0f855d4fd875","Type":"ContainerDied","Data":"bba32dfd1c70cd5e17aaa3b91b7ed28823f860e662ebb7630c40fc4155511cf6"} Oct 29 22:57:15.961254 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:15.961228 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:16.023322 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.023289 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-xks-ca\") pod \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " Oct 29 22:57:16.023528 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.023462 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b9725d-7a83-4ccc-bc79-0f855d4fd875-kube-api-access-s88cj\" (UniqueName: \"kubernetes.io/projected/43b9725d-7a83-4ccc-bc79-0f855d4fd875-kube-api-access-s88cj\") pod \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " Oct 29 22:57:16.023620 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.023603 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-openshift-ca\") pod \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\" (UID: \"43b9725d-7a83-4ccc-bc79-0f855d4fd875\") " Oct 29 22:57:16.024057 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.024027 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-openshift-ca" pod "43b9725d-7a83-4ccc-bc79-0f855d4fd875" (UID: "43b9725d-7a83-4ccc-bc79-0f855d4fd875"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:16.025418 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.025389 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-xks-ca" pod "43b9725d-7a83-4ccc-bc79-0f855d4fd875" (UID: "43b9725d-7a83-4ccc-bc79-0f855d4fd875"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:16.025763 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.025741 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b9725d-7a83-4ccc-bc79-0f855d4fd875-kube-api-access-s88cj" pod "43b9725d-7a83-4ccc-bc79-0f855d4fd875" (UID: "43b9725d-7a83-4ccc-bc79-0f855d4fd875"). InnerVolumeSpecName "kube-api-access-s88cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:16.125453 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.125371 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88cj\" (UniqueName: \"kubernetes.io/projected/43b9725d-7a83-4ccc-bc79-0f855d4fd875-kube-api-access-s88cj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:16.125453 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.125406 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:16.125453 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.125416 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b9725d-7a83-4ccc-bc79-0f855d4fd875-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:16.819484 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.819453 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" Oct 29 22:57:16.819662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.819494 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j" event={"ID":"43b9725d-7a83-4ccc-bc79-0f855d4fd875","Type":"ContainerDied","Data":"65d9b3441154a352fae76bd5123aea6bafbf859222ba39f96c31d4dc733dd456"} Oct 29 22:57:16.819662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:16.819550 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65d9b3441154a352fae76bd5123aea6bafbf859222ba39f96c31d4dc733dd456" Oct 29 22:57:21.148009 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.147966 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg"] Oct 29 22:57:21.148526 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.148510 5736 cpu_manager.go:414] "containerMap: removing container" podUID="43b9725d-7a83-4ccc-bc79-0f855d4fd875" containerName="revoke-keys" Oct 29 22:57:21.148570 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.148527 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b9725d-7a83-4ccc-bc79-0f855d4fd875" containerName="revoke-keys" Oct 29 22:57:21.148649 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.148638 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b9725d-7a83-4ccc-bc79-0f855d4fd875" containerName="revoke-keys" Oct 29 22:57:21.152325 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.152306 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.175833 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.175798 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:57:21.175833 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.175803 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:57:21.179038 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.179012 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg"] Oct 29 22:57:21.190662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.190588 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b0a20e-g79zg\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.190662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.190636 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b0a20e-g79zg\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.190835 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.190693 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftb4\" (UniqueName: \"kubernetes.io/projected/926af2cf-f48c-4361-8457-42b7b2db9ac1-kube-api-access-fftb4\") pod \"maas-api-revoke-sub-c2b0a20e-g79zg\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.291919 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.291882 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b0a20e-g79zg\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.291919 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.291924 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b0a20e-g79zg\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.292148 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.291943 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftb4\" (UniqueName: \"kubernetes.io/projected/926af2cf-f48c-4361-8457-42b7b2db9ac1-kube-api-access-fftb4\") pod \"maas-api-revoke-sub-c2b0a20e-g79zg\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.292632 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.292603 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b0a20e-g79zg\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.294166 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.294142 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b0a20e-g79zg\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.314290 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.314255 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftb4\" (UniqueName: \"kubernetes.io/projected/926af2cf-f48c-4361-8457-42b7b2db9ac1-kube-api-access-fftb4\") pod \"maas-api-revoke-sub-c2b0a20e-g79zg\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.465951 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.465921 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:21.775614 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.775567 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg"] Oct 29 22:57:21.777915 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:21.777887 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926af2cf_f48c_4361_8457_42b7b2db9ac1.slice/crio-6730c359fc66b3f3ccc24d0b7ee9a2c52698e080eb2e68e6e6d27c80fd109016 WatchSource:0}: Error finding container 6730c359fc66b3f3ccc24d0b7ee9a2c52698e080eb2e68e6e6d27c80fd109016: Status 404 returned error can't find the container with id 6730c359fc66b3f3ccc24d0b7ee9a2c52698e080eb2e68e6e6d27c80fd109016 Oct 29 22:57:21.819750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.819195 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k"] Oct 29 22:57:21.824989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.824963 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:21.848861 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.848827 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k"] Oct 29 22:57:21.857820 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.857788 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" event={"ID":"926af2cf-f48c-4361-8457-42b7b2db9ac1","Type":"ContainerStarted","Data":"6730c359fc66b3f3ccc24d0b7ee9a2c52698e080eb2e68e6e6d27c80fd109016"} Oct 29 22:57:21.899788 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.899749 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:21.899940 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.899790 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66gw\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-kube-api-access-f66gw\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:21.899940 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.899899 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:21.900019 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.899950 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:21.900019 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.899992 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:21.900172 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.900100 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:21.900172 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.900164 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:21.900274 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.900259 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:21.900329 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:21.900316 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.001537 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001499 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.001729 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001544 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66gw\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-kube-api-access-f66gw\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.001729 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001603 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.001729 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001630 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.001729 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001669 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.001914 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001781 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.001914 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001845 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.001985 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001939 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.001985 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001963 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.002048 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.001998 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.002282 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.002257 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.002374 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.002291 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.002432 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.002386 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.002860 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.002834 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.004648 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.004620 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.004972 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.004952 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.023641 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.023565 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.024093 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.024070 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66gw\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-kube-api-access-f66gw\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.139525 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.139493 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:22.461953 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.461927 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k"] Oct 29 22:57:22.465071 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:22.465039 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0c30d7_737c_4e13_ad54_4b7f7cb8a7f1.slice/crio-f3aff2d8a6ee9165fad799836958d9208258f40d0c6440805e9fc1f8c04695fe WatchSource:0}: Error finding container f3aff2d8a6ee9165fad799836958d9208258f40d0c6440805e9fc1f8c04695fe: Status 404 returned error can't find the container with id f3aff2d8a6ee9165fad799836958d9208258f40d0c6440805e9fc1f8c04695fe Oct 29 22:57:22.467725 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.467691 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:22.467833 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.467770 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:22.467833 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.467816 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:22.867408 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.867319 5736 generic.go:356] "Generic (PLEG): container finished" podID="926af2cf-f48c-4361-8457-42b7b2db9ac1" containerID="8ac9d633f78932fbcce759dcb2bf5f2a0768a21398984bba2f87e745594d6521" exitCode=0 Oct 29 22:57:22.867611 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.867410 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" event={"ID":"926af2cf-f48c-4361-8457-42b7b2db9ac1","Type":"ContainerDied","Data":"8ac9d633f78932fbcce759dcb2bf5f2a0768a21398984bba2f87e745594d6521"} Oct 29 22:57:22.869439 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.869405 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" event={"ID":"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1","Type":"ContainerStarted","Data":"4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c"} Oct 29 22:57:22.869541 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.869459 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" event={"ID":"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1","Type":"ContainerStarted","Data":"f3aff2d8a6ee9165fad799836958d9208258f40d0c6440805e9fc1f8c04695fe"} Oct 29 22:57:22.950832 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:22.949651 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podStartSLOduration=1.949631128 podStartE2EDuration="1.949631128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:22.948289709 +0000 UTC m=+840.064209316" watchObservedRunningTime="2026-10-29 22:57:22.949631128 +0000 UTC m=+840.065550734" Oct 29 22:57:23.151327 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:23.150723 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:23.151968 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:23.151630 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:57:23.151968 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:23.151689 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:57:24.075499 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.075470 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:24.129786 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.129711 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-openshift-ca\") pod \"926af2cf-f48c-4361-8457-42b7b2db9ac1\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " Oct 29 22:57:24.129924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.129868 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-xks-ca\") pod \"926af2cf-f48c-4361-8457-42b7b2db9ac1\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " Oct 29 22:57:24.130044 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.130025 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926af2cf-f48c-4361-8457-42b7b2db9ac1-kube-api-access-fftb4\" (UniqueName: \"kubernetes.io/projected/926af2cf-f48c-4361-8457-42b7b2db9ac1-kube-api-access-fftb4\") pod \"926af2cf-f48c-4361-8457-42b7b2db9ac1\" (UID: \"926af2cf-f48c-4361-8457-42b7b2db9ac1\") " Oct 29 22:57:24.130130 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.130113 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-openshift-ca" pod "926af2cf-f48c-4361-8457-42b7b2db9ac1" (UID: "926af2cf-f48c-4361-8457-42b7b2db9ac1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:24.130462 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.130438 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:24.159076 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.157248 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:57:24.159217 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.159128 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:57:24.159478 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.159425 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-xks-ca" pod "926af2cf-f48c-4361-8457-42b7b2db9ac1" (UID: "926af2cf-f48c-4361-8457-42b7b2db9ac1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:24.161625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.161569 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926af2cf-f48c-4361-8457-42b7b2db9ac1-kube-api-access-fftb4" pod "926af2cf-f48c-4361-8457-42b7b2db9ac1" (UID: "926af2cf-f48c-4361-8457-42b7b2db9ac1"). InnerVolumeSpecName "kube-api-access-fftb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:24.255675 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.252634 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/926af2cf-f48c-4361-8457-42b7b2db9ac1-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:24.255675 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.252694 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fftb4\" (UniqueName: \"kubernetes.io/projected/926af2cf-f48c-4361-8457-42b7b2db9ac1-kube-api-access-fftb4\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:24.758197 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.758154 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw"] Oct 29 22:57:24.758757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.758732 5736 cpu_manager.go:414] "containerMap: removing container" podUID="926af2cf-f48c-4361-8457-42b7b2db9ac1" containerName="revoke-keys" Oct 29 22:57:24.758757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.758751 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926af2cf-f48c-4361-8457-42b7b2db9ac1" containerName="revoke-keys" Oct 29 22:57:24.758914 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.758876 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="926af2cf-f48c-4361-8457-42b7b2db9ac1" containerName="revoke-keys" Oct 29 22:57:24.762222 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.762199 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.773610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.773560 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw"] Oct 29 22:57:24.861750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.861708 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882db266-6c50-4d21-a39d-1b6459694d44-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.861930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.861768 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.861930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.861885 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882db266-6c50-4d21-a39d-1b6459694d44-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.861930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.861917 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.862042 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.862000 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.862077 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.862048 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.862155 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.862138 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.862191 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.862183 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2m7\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-kube-api-access-nl2m7\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.862237 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.862214 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.885306 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.885278 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" Oct 29 22:57:24.885480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.885316 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg" event={"ID":"926af2cf-f48c-4361-8457-42b7b2db9ac1","Type":"ContainerDied","Data":"6730c359fc66b3f3ccc24d0b7ee9a2c52698e080eb2e68e6e6d27c80fd109016"} Oct 29 22:57:24.885480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.885374 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6730c359fc66b3f3ccc24d0b7ee9a2c52698e080eb2e68e6e6d27c80fd109016" Oct 29 22:57:24.963087 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963050 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882db266-6c50-4d21-a39d-1b6459694d44-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.963292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963103 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.963292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963147 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882db266-6c50-4d21-a39d-1b6459694d44-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.963292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963165 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.963292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963196 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.963292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963235 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.963531 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963318 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.963531 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963339 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2m7\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-kube-api-access-nl2m7\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.963531 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963359 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.963894 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963867 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.964027 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.963946 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.964099 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.964066 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.964237 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.964199 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.964400 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.964375 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882db266-6c50-4d21-a39d-1b6459694d44-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.965872 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.965845 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.965963 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.965936 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882db266-6c50-4d21-a39d-1b6459694d44-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.984642 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.984616 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:24.984855 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:24.984835 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2m7\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-kube-api-access-nl2m7\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:25.078693 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.078617 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:25.158606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.157022 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:57:25.158606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.157078 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:57:25.420324 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.420246 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw"] Oct 29 22:57:25.423678 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:25.423634 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882db266_6c50_4d21_a39d_1b6459694d44.slice/crio-acf4cd00033e956fd7d8bb6921865f84ee9fe8cf48cbc2c2d01ad10203af8bcd WatchSource:0}: Error finding container acf4cd00033e956fd7d8bb6921865f84ee9fe8cf48cbc2c2d01ad10203af8bcd: Status 404 returned error can't find the container with id acf4cd00033e956fd7d8bb6921865f84ee9fe8cf48cbc2c2d01ad10203af8bcd Oct 29 22:57:25.426122 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.426090 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:25.426250 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.426152 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:25.426250 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.426180 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:25.897352 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.897315 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" event={"ID":"882db266-6c50-4d21-a39d-1b6459694d44","Type":"ContainerStarted","Data":"49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139"} Oct 29 22:57:25.897556 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.897391 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" event={"ID":"882db266-6c50-4d21-a39d-1b6459694d44","Type":"ContainerStarted","Data":"acf4cd00033e956fd7d8bb6921865f84ee9fe8cf48cbc2c2d01ad10203af8bcd"} Oct 29 22:57:25.943203 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:25.943118 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podStartSLOduration=1.943098813 podStartE2EDuration="1.943098813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:25.93903053 +0000 UTC m=+843.054950137" watchObservedRunningTime="2026-10-29 22:57:25.943098813 +0000 UTC m=+843.059018422" Oct 29 22:57:26.078921 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:26.078888 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:26.080474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:26.080444 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:57:26.080574 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:26.080492 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 22:57:26.140604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:26.140065 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:57:26.140604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:26.140144 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:57:26.824243 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:57:26.824206 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:57:27.082014 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:27.081928 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:57:27.082476 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:27.082451 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 22:57:27.142033 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:27.141996 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:57:27.144154 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:27.144117 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:57:28.081968 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.081929 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:57:28.082193 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.082007 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 22:57:28.140604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.139943 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:57:28.140604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.140015 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:57:28.293039 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.291592 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2"] Oct 29 22:57:28.298615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.296755 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.307602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.304556 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2"] Oct 29 22:57:28.420082 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.419990 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192952f8-2bda-4164-9e1a-5dbbfbc63017-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.420082 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.420048 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.420082 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.420068 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.420294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.420115 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.420294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.420139 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.420294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.420179 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.420294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.420258 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.420294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.420292 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5b9\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-kube-api-access-sm5b9\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.420449 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.420316 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522054 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522009 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522207 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522125 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522207 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522174 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5b9\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-kube-api-access-sm5b9\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522328 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522210 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522328 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522304 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192952f8-2bda-4164-9e1a-5dbbfbc63017-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522443 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522341 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522443 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522370 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522443 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522426 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522458 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522462 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522473 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522867 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.522997 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.522974 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.523481 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.523455 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192952f8-2bda-4164-9e1a-5dbbfbc63017-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.524920 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.524898 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.525150 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.525133 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.549461 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.548393 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.558900 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.552427 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5b9\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-kube-api-access-sm5b9\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.629701 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.629668 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:28.770601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.770387 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp"] Oct 29 22:57:28.782147 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.778435 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:28.800626 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.799284 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp"] Oct 29 22:57:28.822108 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.821971 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e57216\"/\"distinct-2-w0-e57216-kserve-self-signed-certs\"" Oct 29 22:57:28.927780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.927751 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tmp-dir\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:28.927948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.927786 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-dshm\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:28.927948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.927898 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tls-certs\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:28.928047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.927978 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrpt\" (UniqueName: \"kubernetes.io/projected/0bcce0f2-4dd8-4022-933f-e826e3b07b11-kube-api-access-gsrpt\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:28.928090 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.928068 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-home\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:28.928140 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.928125 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-model-cache\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:28.993290 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:28.993195 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2"] Oct 29 22:57:28.999753 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:28.999723 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192952f8_2bda_4164_9e1a_5dbbfbc63017.slice/crio-c3207d9a49d8a8db536ead552fda494ed6cb2a5304cb4bf8b771441b41348e4c WatchSource:0}: Error finding container c3207d9a49d8a8db536ead552fda494ed6cb2a5304cb4bf8b771441b41348e4c: Status 404 returned error can't find the container with id c3207d9a49d8a8db536ead552fda494ed6cb2a5304cb4bf8b771441b41348e4c Oct 29 22:57:29.002066 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.001695 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:29.002066 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.001777 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:29.002066 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.001830 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:29.029536 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.029502 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tls-certs\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.029695 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.029669 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrpt\" (UniqueName: \"kubernetes.io/projected/0bcce0f2-4dd8-4022-933f-e826e3b07b11-kube-api-access-gsrpt\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.029812 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.029795 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-home\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.029905 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.029888 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-model-cache\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.029961 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.029948 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tmp-dir\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.030003 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.029980 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-dshm\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.031335 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.030852 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-model-cache\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.031335 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.030876 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tmp-dir\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.033615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.033554 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tls-certs\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.037605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.035497 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-home\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.043603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.040474 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-dshm\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.058771 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.058698 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrpt\" (UniqueName: \"kubernetes.io/projected/0bcce0f2-4dd8-4022-933f-e826e3b07b11-kube-api-access-gsrpt\") pod \"distinct-2-w0-e57216-kserve-6c47898746-qrgsp\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.081614 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.080561 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:57:29.081614 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.080636 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 22:57:29.107606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.104604 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:29.152609 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.150007 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:57:29.152609 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.150082 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:57:29.573497 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.573462 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp"] Oct 29 22:57:29.574381 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:29.574350 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bcce0f2_4dd8_4022_933f_e826e3b07b11.slice/crio-ba641eccfae93a428fc79da74f1e7d890b5a09a7e681db7e8842f253afc93702 WatchSource:0}: Error finding container ba641eccfae93a428fc79da74f1e7d890b5a09a7e681db7e8842f253afc93702: Status 404 returned error can't find the container with id ba641eccfae93a428fc79da74f1e7d890b5a09a7e681db7e8842f253afc93702 Oct 29 22:57:29.940375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.940291 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" event={"ID":"0bcce0f2-4dd8-4022-933f-e826e3b07b11","Type":"ContainerStarted","Data":"45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b"} Oct 29 22:57:29.940610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.940569 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" event={"ID":"0bcce0f2-4dd8-4022-933f-e826e3b07b11","Type":"ContainerStarted","Data":"ba641eccfae93a428fc79da74f1e7d890b5a09a7e681db7e8842f253afc93702"} Oct 29 22:57:29.943477 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.943451 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" event={"ID":"192952f8-2bda-4164-9e1a-5dbbfbc63017","Type":"ContainerStarted","Data":"15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6"} Oct 29 22:57:29.943876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:29.943854 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" event={"ID":"192952f8-2bda-4164-9e1a-5dbbfbc63017","Type":"ContainerStarted","Data":"c3207d9a49d8a8db536ead552fda494ed6cb2a5304cb4bf8b771441b41348e4c"} Oct 29 22:57:30.049604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.048745 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podStartSLOduration=2.048728118 podStartE2EDuration="2.048728118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:29.995367771 +0000 UTC m=+847.111287388" watchObservedRunningTime="2026-10-29 22:57:30.048728118 +0000 UTC m=+847.164647726" Oct 29 22:57:30.088604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.087778 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:57:30.088604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.087840 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 22:57:30.108601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.107419 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:30.548602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.548518 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podStartSLOduration=2.548484075 podStartE2EDuration="2.548484075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:30.052746278 +0000 UTC m=+847.168665919" watchObservedRunningTime="2026-10-29 22:57:30.548484075 +0000 UTC m=+847.664403684" Oct 29 22:57:30.555603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.553271 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z"] Oct 29 22:57:30.567720 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.565984 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.608604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.605742 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z"] Oct 29 22:57:30.622189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.621496 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.622189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.621600 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.622189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.621624 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.622189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.621683 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllkc\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-kube-api-access-bllkc\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.622189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.621724 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.622189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.621794 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.622189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.621821 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.622189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.621836 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.622189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.621855 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.630049 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.630022 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:30.707059 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.706722 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:30.707059 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.706786 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:30.722815 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.722780 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.722982 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.722855 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.722982 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.722893 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.723276 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.723251 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.723445 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.723431 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.723563 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.723551 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.723697 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.723682 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.723839 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.723819 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllkc\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-kube-api-access-bllkc\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.723935 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.723614 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.723989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.723324 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.724217 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.724192 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.724301 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.724287 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.724364 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.724325 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.724609 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.724570 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.726080 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.726053 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.726826 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.726802 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.795754 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.795251 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:30.797401 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.796529 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:30.801295 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.800916 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 22:57:30.801295 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.801210 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.802480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.802437 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllkc\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-kube-api-access-bllkc\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:30.929668 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:30.929638 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:31.120616 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.119870 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:57:31.120616 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.119949 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 22:57:31.121351 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.120661 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn"] Oct 29 22:57:31.128899 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.128841 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn"] Oct 29 22:57:31.128899 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.128876 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx"] Oct 29 22:57:31.139689 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.129185 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.167602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.165783 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx"] Oct 29 22:57:31.167602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.165926 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.279684 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.279634 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280109 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280147 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84035ea-6637-46bb-915c-7912e9734d70-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280181 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280211 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280240 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280308 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84035ea-6637-46bb-915c-7912e9734d70-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280381 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2slc\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-kube-api-access-d2slc\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280424 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280460 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280486 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280517 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de92d227-c2af-4a2c-92bc-6c85b4e77056-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.280602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280553 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.281524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280660 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.281524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280710 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.281524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280760 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.281524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280865 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5kt8\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-kube-api-access-s5kt8\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.281524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.280935 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.386662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.386248 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.386662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.386354 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.386662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.386414 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.386662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.386440 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84035ea-6637-46bb-915c-7912e9734d70-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.386662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.386486 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.386662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.386514 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.386662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.386555 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.386662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.386634 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84035ea-6637-46bb-915c-7912e9734d70-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.387213 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387104 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387348 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2slc\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-kube-api-access-d2slc\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387402 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387442 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387449 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387468 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387504 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de92d227-c2af-4a2c-92bc-6c85b4e77056-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387534 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84035ea-6637-46bb-915c-7912e9734d70-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387539 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387669 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387718 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387766 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.387864 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.387855 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kt8\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-kube-api-access-s5kt8\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.388866 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.388428 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.388866 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.388680 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.388866 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.388733 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.389031 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.388927 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.389217 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.389173 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.389403 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.389380 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de92d227-c2af-4a2c-92bc-6c85b4e77056-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.389494 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.389395 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.390483 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.390460 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84035ea-6637-46bb-915c-7912e9734d70-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.390823 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.390805 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.391142 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.391099 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.391730 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.391709 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.409502 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.409462 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2slc\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-kube-api-access-d2slc\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.409860 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.409834 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.409932 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.409917 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5kt8\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-kube-api-access-s5kt8\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.410468 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.410432 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.490433 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.490387 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z"] Oct 29 22:57:31.492944 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:31.492304 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ea4ab6_c8bb_4388_a2a4_44e95490750d.slice/crio-4c8e741b582ef55837bc7af9f17db0dc331d07f25b7c6832e5d1df773b94d672 WatchSource:0}: Error finding container 4c8e741b582ef55837bc7af9f17db0dc331d07f25b7c6832e5d1df773b94d672: Status 404 returned error can't find the container with id 4c8e741b582ef55837bc7af9f17db0dc331d07f25b7c6832e5d1df773b94d672 Oct 29 22:57:31.494813 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.494781 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:31.494905 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.494844 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:31.494905 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.494870 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:31.513153 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.512973 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:31.520619 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.520451 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:31.671370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.666905 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:31.671370 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.666977 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:31.724083 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.724054 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm"] Oct 29 22:57:31.744379 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.743952 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm"] Oct 29 22:57:31.827341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.818885 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6f8123-5t6mm\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:31.827341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.826903 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grd8c\" (UniqueName: \"kubernetes.io/projected/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-kube-api-access-grd8c\") pod \"maas-api-revoke-sub-0c6f8123-5t6mm\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:31.827341 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.827064 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6f8123-5t6mm\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:31.905816 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.865677 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:31.905816 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.897019 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:57:31.905816 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:31.897864 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:57:32.018656 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.018605 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6f8123-5t6mm\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:32.018884 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.018866 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grd8c\" (UniqueName: \"kubernetes.io/projected/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-kube-api-access-grd8c\") pod \"maas-api-revoke-sub-0c6f8123-5t6mm\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:32.019028 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.019013 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6f8123-5t6mm\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:32.020316 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.020294 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6f8123-5t6mm\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:32.023474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.023437 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6f8123-5t6mm\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:32.095495 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.095281 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:57:32.095811 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.095778 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 22:57:32.124769 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.124733 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" event={"ID":"e0ea4ab6-c8bb-4388-a2a4-44e95490750d","Type":"ContainerStarted","Data":"0935a22fd197706b18f506318137970cff11bf752c2f16ad35cfbe3b68e2730b"} Oct 29 22:57:32.125219 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.124828 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" event={"ID":"e0ea4ab6-c8bb-4388-a2a4-44e95490750d","Type":"ContainerStarted","Data":"4c8e741b582ef55837bc7af9f17db0dc331d07f25b7c6832e5d1df773b94d672"} Oct 29 22:57:32.138608 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.130634 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn"] Oct 29 22:57:32.182607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.147621 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:32.182912 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.181356 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grd8c\" (UniqueName: \"kubernetes.io/projected/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-kube-api-access-grd8c\") pod \"maas-api-revoke-sub-0c6f8123-5t6mm\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:32.183056 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.182900 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:32.183189 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.183169 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:32.215194 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.215140 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podStartSLOduration=2.215117719 podStartE2EDuration="2.215117719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:32.210011158 +0000 UTC m=+849.325930763" watchObservedRunningTime="2026-10-29 22:57:32.215117719 +0000 UTC m=+849.331037326" Oct 29 22:57:32.280430 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.280351 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx"] Oct 29 22:57:32.290379 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:32.290348 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde92d227_c2af_4a2c_92bc_6c85b4e77056.slice/crio-8122e2daf74b4d082d477f1cbe33e81eaa82c7ee23698e3831b7534adf5dde34 WatchSource:0}: Error finding container 8122e2daf74b4d082d477f1cbe33e81eaa82c7ee23698e3831b7534adf5dde34: Status 404 returned error can't find the container with id 8122e2daf74b4d082d477f1cbe33e81eaa82c7ee23698e3831b7534adf5dde34 Oct 29 22:57:32.292872 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.292836 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:32.292962 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.292923 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:32.293018 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.292968 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:32.467658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.466817 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:32.705613 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.698374 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:32.705613 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.698432 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:32.963658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.962280 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:32.963658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.963185 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:32.963658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:32.963234 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:33.105531 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.105500 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:57:33.105775 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.105750 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 29 22:57:33.205023 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.204837 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm"] Oct 29 22:57:33.209821 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.209789 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" event={"ID":"bc262d7b-dbd9-4d57-b494-eeafa1cd450b","Type":"ContainerStarted","Data":"f64c86cb1819ab3caab4f879a8a25c7fbad3fbcf9feab6d8cb4c9e9494e3843d"} Oct 29 22:57:33.212638 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.212611 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" event={"ID":"d84035ea-6637-46bb-915c-7912e9734d70","Type":"ContainerStarted","Data":"736421b4c3fc58fa47af1147a92388e1fbfc2b2cd19655bf046c1f79d5b4703e"} Oct 29 22:57:33.214784 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.214735 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" event={"ID":"d84035ea-6637-46bb-915c-7912e9734d70","Type":"ContainerStarted","Data":"489b97dfdd9565c642579b2e41ce280ebdb34d5eff88074c6de9c1598f2528cf"} Oct 29 22:57:33.216983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.216907 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" event={"ID":"de92d227-c2af-4a2c-92bc-6c85b4e77056","Type":"ContainerStarted","Data":"c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c"} Oct 29 22:57:33.216983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.216946 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" event={"ID":"de92d227-c2af-4a2c-92bc-6c85b4e77056","Type":"ContainerStarted","Data":"8122e2daf74b4d082d477f1cbe33e81eaa82c7ee23698e3831b7534adf5dde34"} Oct 29 22:57:33.325646 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.324708 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podStartSLOduration=3.3246878 podStartE2EDuration="3.3246878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:33.279897615 +0000 UTC m=+850.395817224" watchObservedRunningTime="2026-10-29 22:57:33.3246878 +0000 UTC m=+850.440607419" Oct 29 22:57:33.326318 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.326279 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podStartSLOduration=3.326270875 podStartE2EDuration="3.326270875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:33.322911648 +0000 UTC m=+850.438831267" watchObservedRunningTime="2026-10-29 22:57:33.326270875 +0000 UTC m=+850.442190480" Oct 29 22:57:33.524660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.517076 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:33.524660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.517232 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:33.524660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.518423 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:33.524660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.523216 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:33.524660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.523325 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:33.524660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.523367 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:33.642626 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.641732 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:33.642626 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.641795 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:33.933603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.932004 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:33.933603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:33.932068 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:34.260600 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.259646 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" event={"ID":"bc262d7b-dbd9-4d57-b494-eeafa1cd450b","Type":"ContainerStarted","Data":"5b45747476c4b0d7ebdaf0dabef71c584e720e01f690ef9e0b99c0b5a8c8d803"} Oct 29 22:57:34.335609 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.335054 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" podStartSLOduration=3.335031979 podStartE2EDuration="3.335031979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:34.309901836 +0000 UTC m=+851.425821438" watchObservedRunningTime="2026-10-29 22:57:34.335031979 +0000 UTC m=+851.450951589" Oct 29 22:57:34.515447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.514952 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:34.515447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.515017 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:34.531085 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.531051 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:34.531317 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.531293 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:34.577879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.577842 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 22:57:34.600605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.598679 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:34.634683 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.630245 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:34.634683 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.630305 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:34.676603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.676247 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:34.757604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.756696 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:34.782661 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.772841 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:34.803604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.787988 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:34.868605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.860113 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:34.868605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.861342 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:34.868605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.865659 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:34.885056 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.885028 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 22:57:34.894179 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.894144 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:34.948748 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.948704 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:34.948944 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:34.948771 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:35.020800 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.018684 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.042051 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.035662 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.074625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.073672 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.111602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.108921 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.386730 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.382541 5736 generic.go:356] "Generic (PLEG): container finished" podID="bc262d7b-dbd9-4d57-b494-eeafa1cd450b" containerID="5b45747476c4b0d7ebdaf0dabef71c584e720e01f690ef9e0b99c0b5a8c8d803" exitCode=0 Oct 29 22:57:35.386730 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.383178 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" event={"ID":"bc262d7b-dbd9-4d57-b494-eeafa1cd450b","Type":"ContainerDied","Data":"5b45747476c4b0d7ebdaf0dabef71c584e720e01f690ef9e0b99c0b5a8c8d803"} Oct 29 22:57:35.506604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.505721 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.518605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.516663 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:35.518605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.516743 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:35.522603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.521010 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:35.522603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.521061 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:35.593607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.589795 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.613222 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.611233 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.613222 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.611657 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:57:35.650658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.647150 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:35.654718 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.654670 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:35.687711 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.687665 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.688347 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.688323 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:57:35.768022 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.767975 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.768625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.768601 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:57:35.775870 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.775806 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 29 22:57:35.776505 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.776485 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:57:35.831603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.825754 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.831603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.826107 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:57:35.902806 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.902710 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:57:35.918005 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.902978 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:35.941603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.918504 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:57:35.942739 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.903007 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 22:57:35.946599 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.943204 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:57:35.989006 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.977633 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 22:57:35.989518 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:35.978719 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:36.002602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.001119 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:36.024230 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.024171 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:36.030338 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.030308 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:57:36.054309 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.054274 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:36.055480 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.055459 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:57:36.096625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.096556 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:36.110825 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.110755 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:36.111833 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.111813 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:57:36.145447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.140505 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:57:36.145447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.141421 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:57:36.263625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.243212 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:36.527434 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.527341 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:36.528030 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.527998 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:36.528763 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.528738 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:36.528930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.528910 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:36.547179 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.547151 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:57:36.627615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.627281 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:57:36.631635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.583195 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:36.631635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.593667 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 22:57:36.631635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.629451 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:57:36.810719 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.761176 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 22:57:36.810719 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.769947 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:36.810719 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.770460 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:36.810719 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.778656 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 29 22:57:36.975648 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.950847 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:36.975648 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:36.951555 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:37.025521 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.025479 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:57:37.062698 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.049463 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.181900 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.087864 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.181900 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.114125 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 22:57:37.181900 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.114962 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:57:37.280545 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.263310 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:37.280986 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.280951 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:37.281973 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.281939 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 22:57:37.451681 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.451566 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.604984 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.604932 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.605766 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.605730 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:37.668664 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.668621 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:37.737856 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.736859 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:37.767623 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.765558 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.767623 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.767267 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.771618 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.768334 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.787826 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.787789 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.789676 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.789640 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.802960 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.802921 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.845356 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.845312 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.845881 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.845838 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:37.871096 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.869169 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:37.922644 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.872683 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:37.922902 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.904130 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:37.923061 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.923030 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:37.951351 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:37.951308 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 29 22:57:38.070651 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.067884 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:57:38.103478 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.103427 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.113281 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.111281 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:57:38.114550 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.111684 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:57:38.115285 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.111734 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 22:57:38.116051 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.116035 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:57:38.149853 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.149821 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:38.150127 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.150095 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:38.151425 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.151385 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.160383 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.160351 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.172784 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.172751 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:38.309211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.309172 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-xks-ca\") pod \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " Oct 29 22:57:38.311021 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.310987 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-kube-api-access-grd8c\" (UniqueName: \"kubernetes.io/projected/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-kube-api-access-grd8c\") pod \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " Oct 29 22:57:38.311280 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.311258 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-openshift-ca\") pod \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\" (UID: \"bc262d7b-dbd9-4d57-b494-eeafa1cd450b\") " Oct 29 22:57:38.312569 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.312539 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 22:57:38.329775 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.318453 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:57:38.357603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.319976 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-openshift-ca" pod "bc262d7b-dbd9-4d57-b494-eeafa1cd450b" (UID: "bc262d7b-dbd9-4d57-b494-eeafa1cd450b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:38.357903 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.320718 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.391602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.374847 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:57:38.417212 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.417180 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:38.428667 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.428558 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:57:38.587676 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.556669 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.711610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.710267 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:57:38.711610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.711313 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-kube-api-access-grd8c" pod "bc262d7b-dbd9-4d57-b494-eeafa1cd450b" (UID: "bc262d7b-dbd9-4d57-b494-eeafa1cd450b"). InnerVolumeSpecName "kube-api-access-grd8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:38.730045 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.719041 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grd8c\" (UniqueName: \"kubernetes.io/projected/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-kube-api-access-grd8c\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:38.730045 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.719974 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:38.730045 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.720026 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:38.740612 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.739675 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:38.762616 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.760649 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.772607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.768733 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:38.772607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.768804 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:38.772607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.771675 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:57:38.772607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.771678 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.772981 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.772790 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-xks-ca" pod "bc262d7b-dbd9-4d57-b494-eeafa1cd450b" (UID: "bc262d7b-dbd9-4d57-b494-eeafa1cd450b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:38.794610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.793560 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.794610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.793957 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:57:38.803608 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.796089 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.803608 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.796403 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:57:38.810696 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.810658 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.849654 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.827512 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc262d7b-dbd9-4d57-b494-eeafa1cd450b-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:57:38.867606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.850109 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.871639 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.870930 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:38.887710 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.887646 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:38.888091 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.888044 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:38.966621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.944692 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:38.966621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.944752 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:39.000607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:38.971543 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.198659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.181395 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.275649 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.183315 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 29 22:57:39.360605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.334667 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 22:57:39.397954 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.397905 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.457754 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.457641 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 22:57:39.529772 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.529739 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:39.541147 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.541094 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:39.560603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.542059 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:39.560603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.542058 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:39.560603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.554735 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:39.560603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.560371 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.561022 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.560771 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:57:39.642610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.637691 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:39.642610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.637762 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.642610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.642216 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 29 22:57:39.642610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.642262 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 29 22:57:39.755612 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.739620 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.775687 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.745056 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.799613 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.766736 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 29 22:57:39.802856 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.802819 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.803701 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.803673 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.840978 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.780652 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.841448 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.805712 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.841825 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.814902 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" Oct 29 22:57:39.853859 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.814924 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm" event={"ID":"bc262d7b-dbd9-4d57-b494-eeafa1cd450b","Type":"ContainerDied","Data":"f64c86cb1819ab3caab4f879a8a25c7fbad3fbcf9feab6d8cb4c9e9494e3843d"} Oct 29 22:57:39.854040 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.854023 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f64c86cb1819ab3caab4f879a8a25c7fbad3fbcf9feab6d8cb4c9e9494e3843d" Oct 29 22:57:39.892603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.814965 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 22:57:39.892603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.892050 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 22:57:39.896602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.895755 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:39.975605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.973913 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.025137 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.994796 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:40.025137 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:39.994851 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:40.189618 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.185656 5736 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded" Oct 29 22:57:40.218605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.207881 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.284686 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.249069 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 22:57:40.285240 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.284614 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.364643 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.344664 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.365711 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.365668 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.399424 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.399377 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.400113 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.400087 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:57:40.468705 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.463761 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.468705 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.464303 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:57:40.536659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.513854 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:40.536659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.513913 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:40.584611 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.573016 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:40.595747 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.595682 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:40.603602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.574675 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.700299 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.700250 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.796475 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.796361 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:40.833028 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.805658 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 29 22:57:40.861993 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.831852 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 22:57:40.862512 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.858877 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 29 22:57:40.862917 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.858911 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 22:57:40.874925 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.858950 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 22:57:40.875356 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.862617 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 29 22:57:40.894603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.863096 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:57:40.897525 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.895605 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 22:57:40.897706 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.897258 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.914739 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.914703 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:40.948656 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.944166 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:57:40.949038 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.948991 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:40.949267 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.949232 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:40.981720 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:40.981666 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.085665 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.048055 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.155888 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.155761 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.342664 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.341789 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.429612 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.393644 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 29 22:57:41.507605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.431121 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:57:41.562612 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.434937 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.562612 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.562237 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:57:41.562849 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.447218 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.562912 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.514351 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:41.562960 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.562931 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:41.563006 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.539791 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:41.563006 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.562982 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:41.563086 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.511919 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.621660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.612689 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.754610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.738738 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.754610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.738893 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:41.754610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.738938 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.848604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.836644 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:41.862694 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.862652 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 22:57:41.866919 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.862903 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.874656 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.862938 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 29 22:57:41.899659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.874852 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.900305 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.900266 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.900753 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.900728 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.953604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.946063 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:41.976604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.954156 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 22:57:41.995604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.971791 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:42.003604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.999656 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:42.003869 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:41.994717 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.243612 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.239666 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 29 22:57:42.359615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.356688 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.402775 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.402726 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.530028 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.529933 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:42.530279 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.530247 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:42.530729 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.530710 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:42.530876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.530856 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:42.576466 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.574791 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.576466 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.575571 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.576466 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.575857 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.576466 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.576084 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:57:42.577230 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.577188 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.613745 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.613701 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.788657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.783161 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:42.788657 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.783247 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.871688 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.871643 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:42.873958 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.872233 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.873958 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.872276 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.878330 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.877650 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.904711 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.902684 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.904711 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.903064 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 22:57:42.904711 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.903301 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 29 22:57:42.933032 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.932913 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:42.933032 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.932970 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:42.954863 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.954666 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:42.994750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:42.994708 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 22:57:43.009016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.008234 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.080697 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.080541 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.199039 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.199004 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:43.199477 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.199357 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.212521 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.212490 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:43.212754 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.212724 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.363655 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.362371 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.404039 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.403667 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 29 22:57:43.546605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.530146 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:43.546605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.530205 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:43.546605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.530941 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:43.546605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.530984 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:43.575001 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.574964 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 22:57:43.575726 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.575696 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:57:43.575965 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.575944 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 22:57:43.576324 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.576300 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:57:43.614746 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.614475 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.629822 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.629790 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:43.790725 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.790687 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:43.792365 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.792315 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.807968 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.807941 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:43.830013 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.829977 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:57:43.887384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.878057 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.924899 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.883228 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:43.928091 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.883323 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 29 22:57:43.928462 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.884358 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 22:57:43.928821 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.903878 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:43.929164 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.903912 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 22:57:43.929489 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.903940 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 22:57:43.939639 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.939609 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:43.939825 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.939800 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:43.956817 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:43.956777 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.007247 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.007204 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 22:57:44.009150 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.009118 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.129731 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.111020 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:57:44.152930 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.152854 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 22:57:44.220385 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.219856 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:57:44.266807 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.266782 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:57:44.341599 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.337414 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.345917 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.345894 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:57:44.406382 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.406292 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.488269 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.474093 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:57:44.529610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.529032 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:44.529610 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.529209 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:44.533987 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.533006 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:44.533987 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.533047 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:44.578012 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.576557 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.581507 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.578423 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.581507 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.580145 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:44.581507 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.580392 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 22:57:44.603857 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.603826 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:57:44.787532 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.787505 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:57:44.999483 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:44.999435 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 22:57:45.022288 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.022252 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:57:45.022565 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.022535 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 29 22:57:45.046078 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.045977 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.046667 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.046634 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:45.048068 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.048042 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 22:57:45.048533 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.048510 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 22:57:45.149624 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.124104 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:57:45.174671 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.174636 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.528730 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.528695 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:45.528986 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.528953 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:45.529570 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.529545 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:45.529737 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.529716 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:45.589198 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.589154 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.589915 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.589566 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:57:45.627716 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.627665 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.639695 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.639658 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2dlnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:45.639952 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.639923 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2dlnz" podUID="98a25bcb-0734-4d38-a889-214cfcf54e8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.869833 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.867756 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:45.883786 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.883738 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 29 22:57:45.970605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.944628 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:46.007615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:45.983829 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 22:57:46.007615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.002695 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.049896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.049853 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 22:57:46.050910 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.050146 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 22:57:46.051307 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.050190 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.051667 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.050212 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.180859 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.180736 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.256603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.250668 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.256603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.251852 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.422605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.419143 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.522603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.520947 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:46.522603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.521028 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:46.522603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.521329 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:46.522603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.521467 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:46.625633 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.619866 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.625633 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.620492 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.625633 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.620615 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.664920 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.664845 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.693637 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.689863 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:57:46.715602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.711691 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:46.715602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.711752 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:46.868848 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.868734 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:46.895606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.869964 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:57:46.983603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.887821 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.009608 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.984346 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:57:47.010284 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:46.986559 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.010473 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.009737 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 22:57:47.012175 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.012130 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:57:47.140733 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.031333 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.141333 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.032058 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:47.141534 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.141504 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.147615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.063551 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 22:57:47.148100 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.083712 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 22:57:47.148546 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.083748 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 22:57:47.158103 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.126681 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.158461 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.083771 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:57:47.182100 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.181236 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.312605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.309109 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 22:57:47.440879 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.440761 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.472429 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.387812 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.764606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.749729 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:47.764606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.749808 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:47.764606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.749915 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:47.764606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.749962 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:47.776771 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.776733 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.792929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.777707 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.792929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.777757 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.792929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.777946 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:57:47.792929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.792467 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:57:47.792929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.792519 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:57:47.792929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.792552 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 22:57:47.792929 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.792617 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:57:47.830116 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.830068 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.897744 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.897696 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:47.984755 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:47.984716 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:48.006451 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.006421 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:57:48.016362 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.016283 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:48.023909 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.018652 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 29 22:57:48.023909 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.022224 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:57:48.023909 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.022276 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:57:48.094155 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.094110 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 29 22:57:48.102592 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.101332 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:57:48.108658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.108636 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:57:48.109353 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.109330 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:57:48.143137 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.128292 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:57:48.143137 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.129274 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:57:48.143137 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.142515 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 22:57:48.159396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.158113 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:48.159396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.158693 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:48.159396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.159240 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:57:48.182315 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.182057 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:48.194948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.194919 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:48.195168 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.195148 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:48.215817 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.215795 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:57:48.230010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.229987 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:57:48.233233 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.233215 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:57:48.233415 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.233404 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:57:48.439507 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.439419 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 22:57:48.517105 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.517070 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:57:48.517385 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.517354 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 29 22:57:48.522088 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.522058 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:57:48.605439 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.605401 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:57:48.605720 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.605676 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 29 22:57:48.831123 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:48.831080 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:49.037108 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.037075 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 22:57:49.094945 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.094848 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:57:49.201036 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.200661 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:49.201036 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.200725 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:49.218238 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.216942 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:49.220940 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.220910 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:49.222665 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.222647 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 22:57:49.335267 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.335203 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:57:49.821906 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.821121 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l"] Oct 29 22:57:49.821906 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.821720 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bc262d7b-dbd9-4d57-b494-eeafa1cd450b" containerName="revoke-keys" Oct 29 22:57:49.821906 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.821737 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc262d7b-dbd9-4d57-b494-eeafa1cd450b" containerName="revoke-keys" Oct 29 22:57:49.821906 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.821838 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc262d7b-dbd9-4d57-b494-eeafa1cd450b" containerName="revoke-keys" Oct 29 22:57:49.826949 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.826926 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.878983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.878952 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l"] Oct 29 22:57:49.928950 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.927879 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:49.929712 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.929648 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.929712 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.929699 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.929913 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.929743 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f61bd156-2fc9-427e-9ae6-224a79565512-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.929913 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.929856 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9c7z\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-kube-api-access-w9c7z\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.930023 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.930000 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.930076 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.930025 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.930076 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.930064 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.930185 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.930165 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.930239 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.930201 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f61bd156-2fc9-427e-9ae6-224a79565512-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:49.930606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:49.930502 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 22:57:50.031612 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.031540 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.031802 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.031625 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.031802 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.031689 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.031802 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.031791 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.031984 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.031828 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f61bd156-2fc9-427e-9ae6-224a79565512-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.031984 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.031865 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.031984 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.031900 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.031984 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.031946 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f61bd156-2fc9-427e-9ae6-224a79565512-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.032172 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.032111 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9c7z\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-kube-api-access-w9c7z\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.032900 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.032872 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.033055 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.033035 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.033910 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.033881 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f61bd156-2fc9-427e-9ae6-224a79565512-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.034411 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.034382 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.034687 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.034664 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.035309 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.035283 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.039492 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.039464 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f61bd156-2fc9-427e-9ae6-224a79565512-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.055071 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.055043 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9c7z\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-kube-api-access-w9c7z\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.055382 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.055357 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.214359 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.214321 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:50.555826 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.555730 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:50.555826 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.555806 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:50.575625 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.575558 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l"] Oct 29 22:57:50.581504 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:50.581475 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61bd156_2fc9_427e_9ae6_224a79565512.slice/crio-843a5a143cd05e34ef6b386295a8846744ecafd33dae6d62111231f9d87f9f1b WatchSource:0}: Error finding container 843a5a143cd05e34ef6b386295a8846744ecafd33dae6d62111231f9d87f9f1b: Status 404 returned error can't find the container with id 843a5a143cd05e34ef6b386295a8846744ecafd33dae6d62111231f9d87f9f1b Oct 29 22:57:50.585156 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.585107 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:50.585270 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.585205 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:50.585270 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.585250 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:50.906134 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.906035 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" event={"ID":"f61bd156-2fc9-427e-9ae6-224a79565512","Type":"ContainerStarted","Data":"16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f"} Oct 29 22:57:50.906134 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.906088 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" event={"ID":"f61bd156-2fc9-427e-9ae6-224a79565512","Type":"ContainerStarted","Data":"843a5a143cd05e34ef6b386295a8846744ecafd33dae6d62111231f9d87f9f1b"} Oct 29 22:57:50.983650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.983622 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:50.987322 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:50.987300 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 22:57:51.027614 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:51.027517 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podStartSLOduration=2.027461264 podStartE2EDuration="2.027461264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:50.959413281 +0000 UTC m=+868.075332887" watchObservedRunningTime="2026-10-29 22:57:51.027461264 +0000 UTC m=+868.143380871" Oct 29 22:57:51.215394 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:51.215363 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:51.216790 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:51.216766 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:51.216874 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:51.216819 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 29 22:57:52.214902 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:52.214863 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:52.215124 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:52.214927 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 29 22:57:53.215034 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:53.214999 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:53.215422 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:53.215053 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 29 22:57:54.215563 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:54.215529 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:54.215983 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:54.215618 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 29 22:57:55.215285 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:55.215248 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:55.215433 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:55.215303 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 29 22:57:56.214880 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:56.214846 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:56.215351 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:56.214907 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 29 22:57:56.825909 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:57:56.825875 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:57:57.215142 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:57.215111 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:57:57.215518 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:57.215163 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 29 22:57:58.414649 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.414609 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:58.415143 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.414969 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:58.415747 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.415729 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 22:57:58.513544 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.513508 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx"] Oct 29 22:57:58.519239 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.519214 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.537804 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.537777 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 22:57:58.537804 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.537782 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 22:57:58.538011 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.537848 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7dsjj\"" Oct 29 22:57:58.545768 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.545734 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx"] Oct 29 22:57:58.553046 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.553016 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ba2b1209-5252-4af5-9ba1-e47404585d06-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.553218 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.553081 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.553218 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.553169 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba2b1209-5252-4af5-9ba1-e47404585d06-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.553347 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.553304 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.553405 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.553367 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.553462 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.553405 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.553564 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.553541 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27gmb\" (UniqueName: \"kubernetes.io/projected/ba2b1209-5252-4af5-9ba1-e47404585d06-kube-api-access-27gmb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.640413 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.640131 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 22:57:58.655095 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.655066 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ba2b1209-5252-4af5-9ba1-e47404585d06-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.655237 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.655117 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.655237 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.655138 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba2b1209-5252-4af5-9ba1-e47404585d06-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.655237 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.655179 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.655237 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.655212 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.655461 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.655237 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.655461 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.655272 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27gmb\" (UniqueName: \"kubernetes.io/projected/ba2b1209-5252-4af5-9ba1-e47404585d06-kube-api-access-27gmb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.656047 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.656015 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.657708 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.657682 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba2b1209-5252-4af5-9ba1-e47404585d06-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.657708 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.657707 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ba2b1209-5252-4af5-9ba1-e47404585d06-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.657851 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.657799 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.657904 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.657885 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.676314 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.676259 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba2b1209-5252-4af5-9ba1-e47404585d06-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.676469 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.676451 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27gmb\" (UniqueName: \"kubernetes.io/projected/ba2b1209-5252-4af5-9ba1-e47404585d06-kube-api-access-27gmb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p6cwx\" (UID: \"ba2b1209-5252-4af5-9ba1-e47404585d06\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:58.834395 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:58.834367 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:57:59.165329 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:59.165295 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx"] Oct 29 22:57:59.169120 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:57:59.169081 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2b1209_5252_4af5_9ba1_e47404585d06.slice/crio-b27d94503151d578d4ab9b263d889a2f38b13f8f12ad7b8cf0b5d42bdd3b76eb WatchSource:0}: Error finding container b27d94503151d578d4ab9b263d889a2f38b13f8f12ad7b8cf0b5d42bdd3b76eb: Status 404 returned error can't find the container with id b27d94503151d578d4ab9b263d889a2f38b13f8f12ad7b8cf0b5d42bdd3b76eb Oct 29 22:57:59.171531 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:59.171499 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:59.171624 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:57:59.171594 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:00.001722 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:00.001673 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" event={"ID":"ba2b1209-5252-4af5-9ba1-e47404585d06","Type":"ContainerStarted","Data":"90df2a6c47a6fe49a0546dda4d4330c0851ee92ef0bc1fad2739bf59fc065899"} Oct 29 22:58:00.002319 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:00.002293 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:58:00.002438 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:00.002331 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" event={"ID":"ba2b1209-5252-4af5-9ba1-e47404585d06","Type":"ContainerStarted","Data":"b27d94503151d578d4ab9b263d889a2f38b13f8f12ad7b8cf0b5d42bdd3b76eb"} Oct 29 22:58:00.003685 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:00.003657 5736 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-p6cwx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:58:00.003827 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:00.003715 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" podUID="ba2b1209-5252-4af5-9ba1-e47404585d06" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:58:00.063178 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:00.063108 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" podStartSLOduration=2.063083602 podStartE2EDuration="2.063083602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:00.047168444 +0000 UTC m=+877.163088085" watchObservedRunningTime="2026-10-29 22:58:00.063083602 +0000 UTC m=+877.179003209" Oct 29 22:58:01.009387 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:01.009345 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p6cwx" Oct 29 22:58:07.040126 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.040087 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd"] Oct 29 22:58:07.044698 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.044674 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.069456 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.069428 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd"] Oct 29 22:58:07.181451 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.181418 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.181650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.181530 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.181650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.181563 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.181650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.181632 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.181792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.181758 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.181828 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.181820 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786b0ea4-d2c8-458e-a329-c65354543981-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.181874 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.181863 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786b0ea4-d2c8-458e-a329-c65354543981-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.181910 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.181891 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn46k\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-kube-api-access-gn46k\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.181947 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.181920 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.282844 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.282815 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.282985 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.282965 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283031 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.282984 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283031 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283004 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283106 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283069 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283106 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283103 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786b0ea4-d2c8-458e-a329-c65354543981-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283211 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283135 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786b0ea4-d2c8-458e-a329-c65354543981-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283296 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283264 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn46k\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-kube-api-access-gn46k\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283359 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283293 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283465 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283441 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283548 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283445 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283548 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283490 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283739 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.283942 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.283915 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786b0ea4-d2c8-458e-a329-c65354543981-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.285311 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.285291 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.285515 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.285499 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786b0ea4-d2c8-458e-a329-c65354543981-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.304463 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.304404 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn46k\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-kube-api-access-gn46k\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.304641 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.304623 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.359786 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.359757 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:07.683178 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.683154 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd"] Oct 29 22:58:07.685451 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:58:07.685419 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786b0ea4_d2c8_458e_a329_c65354543981.slice/crio-af95af4295f3528802396bf2da3c49f9543065f012eea4c5b85ba4bada0afdd9 WatchSource:0}: Error finding container af95af4295f3528802396bf2da3c49f9543065f012eea4c5b85ba4bada0afdd9: Status 404 returned error can't find the container with id af95af4295f3528802396bf2da3c49f9543065f012eea4c5b85ba4bada0afdd9 Oct 29 22:58:07.688010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.687978 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:07.688093 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.688041 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:07.688093 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:07.688071 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:08.060390 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:08.060342 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" event={"ID":"786b0ea4-d2c8-458e-a329-c65354543981","Type":"ContainerStarted","Data":"d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4"} Oct 29 22:58:08.060921 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:08.060414 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" event={"ID":"786b0ea4-d2c8-458e-a329-c65354543981","Type":"ContainerStarted","Data":"af95af4295f3528802396bf2da3c49f9543065f012eea4c5b85ba4bada0afdd9"} Oct 29 22:58:08.118417 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:08.118347 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podStartSLOduration=1.118326313 podStartE2EDuration="1.118326313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:08.115011113 +0000 UTC m=+885.230930722" watchObservedRunningTime="2026-10-29 22:58:08.118326313 +0000 UTC m=+885.234245920" Oct 29 22:58:08.360644 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:08.360551 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:08.362107 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:08.362081 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:58:08.362232 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:08.362141 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 29 22:58:09.360326 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:09.360287 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:58:09.360819 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:09.360350 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 29 22:58:10.067549 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.067512 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh"] Oct 29 22:58:10.072656 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.072605 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.110135 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.110091 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh"] Oct 29 22:58:10.229546 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.229505 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.229713 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.229685 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.229909 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.229875 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.230049 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.229922 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.230049 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.229971 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.230167 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.230120 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs9gc\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-kube-api-access-vs9gc\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.230250 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.230233 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.230289 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.230270 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.230332 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.230308 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.352648 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.331393 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.352969 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.352942 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.353231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.353212 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.353449 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.353426 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.353596 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.353558 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.353738 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.353719 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.353876 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.353860 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs9gc\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-kube-api-access-vs9gc\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.354034 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.354018 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.354152 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.354137 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.354648 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.354627 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.356702 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.355004 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.356702 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.356630 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.356702 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.355875 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.356702 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.342028 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.357168 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.357144 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.358592 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.358559 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.360572 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.360554 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:58:10.360862 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.360609 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 29 22:58:10.376922 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.376902 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.377238 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.377217 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs9gc\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-kube-api-access-vs9gc\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.389613 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.389544 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:10.758340 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.758305 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh"] Oct 29 22:58:10.760934 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:58:10.760900 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58f3e0e_f2c6_4206_b93a_25b3d7e0ffb3.slice/crio-cba8a90b8be81709a0eec256abbdc5b9c279ccaa1248f0e834874a0a63a08ab2 WatchSource:0}: Error finding container cba8a90b8be81709a0eec256abbdc5b9c279ccaa1248f0e834874a0a63a08ab2: Status 404 returned error can't find the container with id cba8a90b8be81709a0eec256abbdc5b9c279ccaa1248f0e834874a0a63a08ab2 Oct 29 22:58:10.764769 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.764734 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:10.764883 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.764808 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:10.764883 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:10.764854 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:11.088684 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:11.088604 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" event={"ID":"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3","Type":"ContainerStarted","Data":"9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7"} Oct 29 22:58:11.088684 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:11.088666 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" event={"ID":"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3","Type":"ContainerStarted","Data":"cba8a90b8be81709a0eec256abbdc5b9c279ccaa1248f0e834874a0a63a08ab2"} Oct 29 22:58:11.153606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:11.153372 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podStartSLOduration=1.153352537 podStartE2EDuration="1.153352537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:11.12882426 +0000 UTC m=+888.244743881" watchObservedRunningTime="2026-10-29 22:58:11.153352537 +0000 UTC m=+888.269272144" Oct 29 22:58:11.360988 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:11.360856 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:58:11.360988 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:11.360918 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 29 22:58:11.390221 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:11.390189 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:11.391572 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:11.391545 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:58:11.391707 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:11.391606 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:58:12.371602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:12.370983 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:58:12.371602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:12.371043 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 29 22:58:12.392608 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:12.392503 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:58:12.392608 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:12.392560 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:58:13.361953 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:13.361915 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:58:13.362233 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:13.362202 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 29 22:58:13.402185 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:13.402146 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:58:13.402798 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:13.402762 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:58:14.363017 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:14.362981 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:58:14.363295 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:14.363256 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 29 22:58:14.390051 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:14.390017 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:58:14.390365 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:14.390320 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:58:15.363074 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:15.363039 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:58:15.363735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:15.363690 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 29 22:58:15.392011 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:15.391977 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:58:15.392258 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:15.392223 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:58:15.853877 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:15.853836 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.032513 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.010008 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.127553 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.126732 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.128981 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.126671 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.130350 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.130322 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 22:58:16.155142 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.155105 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 22:58:16.226396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.225361 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.237602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.233792 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.330606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.247917 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.331722 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.248908 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded" Oct 29 22:58:16.360532 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.360498 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.360829 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.360785 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 29 22:58:16.410756 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.410661 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.411319 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.411280 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:58:16.640200 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.640146 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.759603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.755958 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 22:58:16.863746 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.863695 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.869616 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.868882 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 22:58:16.869616 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.864793 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.873334 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.867749 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.876005 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.867963 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 22:58:16.880724 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.880695 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:16.882112 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.882093 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:58:16.908317 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:16.908279 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.071844 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.071745 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.155745 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.155698 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.157881 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.157840 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.161255 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.161220 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.178573 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.161555 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.305978 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.305947 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:58:17.306326 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.306309 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:58:17.306502 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.306489 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:58:17.310842 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.310726 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:58:17.314112 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.311031 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:58:17.314287 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.314272 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 22:58:17.315674 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.315628 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.321324 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.321304 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:58:17.332939 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.332860 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.334886 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.334732 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:58:17.362277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.360684 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.393604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.386699 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:58:17.393604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.387666 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.464885 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.464849 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:58:17.465381 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.464912 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:58:17.587895 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.587796 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.592420 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.592380 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.789706 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.788671 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.813607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.812994 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.859475 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.858756 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:58:17.859475 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.858821 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:58:17.888735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.870691 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:58:17.888735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.871863 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 22:58:17.888735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.872177 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:58:17.888735 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.872229 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:58:17.890811 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.890769 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.900396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.891032 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:17.900396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.892638 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 22:58:17.900396 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.899637 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:58:17.969153 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:17.969098 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.011601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.011555 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:58:18.106329 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.106283 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 29 22:58:18.162013 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.161919 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.184040 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.162552 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.342959 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.170648 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.343437 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.256205 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 22:58:18.343811 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.326404 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.344120 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.334493 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.347901 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.334935 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:18.363608 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.359886 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:18.369664 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.369636 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 29 22:58:18.417652 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.414844 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:58:18.417652 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.414907 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:58:18.540320 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.540285 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 22:58:18.750602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.740414 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 22:58:18.750602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.741396 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 22:58:18.752288 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.752265 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 22:58:18.815598 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.811812 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.834095 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.834057 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.897296 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.897224 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:18.900021 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.899171 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 22:58:18.900331 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.900315 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 22:58:18.900560 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.900548 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 22:58:18.991500 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.991469 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 22:58:18.992097 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.992073 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 22:58:18.992646 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.992627 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 22:58:18.994666 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.994647 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 22:58:18.998702 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:18.998684 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 22:58:19.027428 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:19.027348 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 22:58:19.027428 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:19.027414 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 22:58:19.028084 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:19.027444 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 22:58:19.390571 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:19.390475 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 29 22:58:19.390967 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:19.390567 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 29 22:58:20.394469 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:20.394440 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:20.394869 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:20.394793 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:20.395440 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:20.395424 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 22:58:23.126024 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:23.125999 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:58:23.127287 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:23.127264 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 22:58:24.394435 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.394393 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5"] Oct 29 22:58:24.399948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.399929 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.419215 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.419184 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5"] Oct 29 22:58:24.425294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.425272 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:58:24.425496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.425479 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:58:24.459444 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.459405 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j74c\" (UniqueName: \"kubernetes.io/projected/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-kube-api-access-2j74c\") pod \"maas-api-revoke-sub-add02620-jjqs5\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.459444 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.459446 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add02620-jjqs5\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.459687 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.459477 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add02620-jjqs5\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.561054 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.561023 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j74c\" (UniqueName: \"kubernetes.io/projected/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-kube-api-access-2j74c\") pod \"maas-api-revoke-sub-add02620-jjqs5\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.561205 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.561063 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add02620-jjqs5\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.561205 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.561092 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add02620-jjqs5\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.561899 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.561880 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add02620-jjqs5\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.563496 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.563468 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add02620-jjqs5\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.584603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.584568 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j74c\" (UniqueName: \"kubernetes.io/projected/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-kube-api-access-2j74c\") pod \"maas-api-revoke-sub-add02620-jjqs5\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:24.713463 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:24.713431 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:25.021612 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:25.021568 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5"] Oct 29 22:58:25.023795 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:58:25.023765 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bd9e21_dcf5_4fbe_a37a_0e2cdb04b1e7.slice/crio-40d703f5a34ed1687662d68306b29f95ef7e92665a9e7da3796afc6f8f19810f WatchSource:0}: Error finding container 40d703f5a34ed1687662d68306b29f95ef7e92665a9e7da3796afc6f8f19810f: Status 404 returned error can't find the container with id 40d703f5a34ed1687662d68306b29f95ef7e92665a9e7da3796afc6f8f19810f Oct 29 22:58:25.802477 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:25.802435 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" event={"ID":"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7","Type":"ContainerStarted","Data":"e74575280fa2b07691c1301c205bb97b5f451fd1a69c5a981c131a80a6a124ed"} Oct 29 22:58:25.802886 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:25.802504 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" event={"ID":"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7","Type":"ContainerStarted","Data":"40d703f5a34ed1687662d68306b29f95ef7e92665a9e7da3796afc6f8f19810f"} Oct 29 22:58:25.840889 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:25.840831 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" podStartSLOduration=1.840812629 podStartE2EDuration="1.840812629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:25.840503723 +0000 UTC m=+902.956423335" watchObservedRunningTime="2026-10-29 22:58:25.840812629 +0000 UTC m=+902.956732236" Oct 29 22:58:26.811846 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:26.811807 5736 generic.go:356] "Generic (PLEG): container finished" podID="39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7" containerID="e74575280fa2b07691c1301c205bb97b5f451fd1a69c5a981c131a80a6a124ed" exitCode=0 Oct 29 22:58:26.812214 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:26.811888 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" event={"ID":"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7","Type":"ContainerDied","Data":"e74575280fa2b07691c1301c205bb97b5f451fd1a69c5a981c131a80a6a124ed"} Oct 29 22:58:26.824135 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:58:26.824109 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:58:27.989770 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:27.989747 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:28.115261 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.115179 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-openshift-ca\") pod \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " Oct 29 22:58:28.115261 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.115253 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-kube-api-access-2j74c\" (UniqueName: \"kubernetes.io/projected/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-kube-api-access-2j74c\") pod \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " Oct 29 22:58:28.115442 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.115303 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-xks-ca\") pod \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\" (UID: \"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7\") " Oct 29 22:58:28.115655 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.115626 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-openshift-ca" pod "39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7" (UID: "39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:28.116190 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.116161 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:28.117425 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.117401 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-kube-api-access-2j74c" pod "39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7" (UID: "39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7"). InnerVolumeSpecName "kube-api-access-2j74c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:28.117541 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.117463 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-xks-ca" pod "39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7" (UID: "39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:28.217644 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.217616 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j74c\" (UniqueName: \"kubernetes.io/projected/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-kube-api-access-2j74c\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:28.217644 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.217642 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:28.828840 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.828808 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" Oct 29 22:58:28.828840 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.828831 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5" event={"ID":"39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7","Type":"ContainerDied","Data":"40d703f5a34ed1687662d68306b29f95ef7e92665a9e7da3796afc6f8f19810f"} Oct 29 22:58:28.829098 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:28.828877 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40d703f5a34ed1687662d68306b29f95ef7e92665a9e7da3796afc6f8f19810f" Oct 29 22:58:35.061336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.061240 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc"] Oct 29 22:58:35.061935 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.061913 5736 cpu_manager.go:414] "containerMap: removing container" podUID="39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7" containerName="revoke-keys" Oct 29 22:58:35.061935 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.061934 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7" containerName="revoke-keys" Oct 29 22:58:35.062070 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.062036 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7" containerName="revoke-keys" Oct 29 22:58:35.066898 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.066877 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.092502 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.092479 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc"] Oct 29 22:58:35.217602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.214544 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwrv6\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-kube-api-access-pwrv6\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.217602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.214644 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.217602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.214735 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.217602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.214801 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.217602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.214837 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.217602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.214905 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.217602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.215019 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.217602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.215062 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.217602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.215137 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.316926 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.316837 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.316926 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.316886 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317169 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.316946 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317169 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317050 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317169 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317092 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317169 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317167 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317382 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317240 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwrv6\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-kube-api-access-pwrv6\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317382 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317279 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317382 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317352 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317933 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317677 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317933 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317709 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.317933 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317864 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.318118 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.317982 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.318232 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.318209 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.319826 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.319799 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.319945 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.319856 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.339846 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.339814 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwrv6\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-kube-api-access-pwrv6\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.339968 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.339921 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.381175 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.381150 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:35.724471 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.724446 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc"] Oct 29 22:58:35.726928 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:58:35.726901 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fab7da7_6892_4bd6_976d_7e8bfc3efc13.slice/crio-d561b74dd824e5fe35bc6e263635e8f9e02408dd938d87e0169eee02c0a3d193 WatchSource:0}: Error finding container d561b74dd824e5fe35bc6e263635e8f9e02408dd938d87e0169eee02c0a3d193: Status 404 returned error can't find the container with id d561b74dd824e5fe35bc6e263635e8f9e02408dd938d87e0169eee02c0a3d193 Oct 29 22:58:35.729431 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.729399 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:35.729524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.729462 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:35.729524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.729490 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:35.891602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.891419 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" event={"ID":"7fab7da7-6892-4bd6-976d-7e8bfc3efc13","Type":"ContainerStarted","Data":"6084ba711f9bf011bd44a2344821a26ab04fccc57ce21888c03772fe26face17"} Oct 29 22:58:35.891602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.891485 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" event={"ID":"7fab7da7-6892-4bd6-976d-7e8bfc3efc13","Type":"ContainerStarted","Data":"d561b74dd824e5fe35bc6e263635e8f9e02408dd938d87e0169eee02c0a3d193"} Oct 29 22:58:35.939863 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:35.939487 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podStartSLOduration=0.939467694 podStartE2EDuration="939.467694ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:35.933235585 +0000 UTC m=+913.049155191" watchObservedRunningTime="2026-10-29 22:58:35.939467694 +0000 UTC m=+913.055387301" Oct 29 22:58:36.381461 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:36.381432 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:36.382978 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:36.382946 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:58:36.383096 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:36.383030 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:58:37.382557 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:37.382522 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:58:37.382957 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:37.382618 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:58:38.381636 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.381600 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:58:38.381943 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.381669 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:58:38.659683 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.659050 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z"] Oct 29 22:58:38.664420 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.664390 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.680204 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.680175 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd"] Oct 29 22:58:38.684858 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.684837 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.687672 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.687646 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z"] Oct 29 22:58:38.720628 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.718860 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd"] Oct 29 22:58:38.788495 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.788453 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.788690 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.788512 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.788690 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.788555 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.788690 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.788675 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.788839 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.788726 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.788839 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.788803 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.788950 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.788886 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmqv\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-kube-api-access-xnmqv\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.788950 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.788937 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z9dn\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-kube-api-access-5z9dn\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.789057 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.788974 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.789119 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.789077 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.789179 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.789142 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.789267 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.789249 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.789431 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.789404 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.789631 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.789610 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.789804 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.789788 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.789965 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.789949 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.790120 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.790103 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.790266 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.790249 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.891150 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891118 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.891340 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891171 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.891340 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891207 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.891340 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891223 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.891340 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891245 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.891589 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891530 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.891654 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891600 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.891654 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891613 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.891766 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891655 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.891766 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891708 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmqv\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-kube-api-access-xnmqv\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.891766 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891761 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z9dn\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-kube-api-access-5z9dn\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.891961 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891795 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.891961 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891838 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.891961 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891893 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.891961 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891945 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.892190 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.891979 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.892190 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892007 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.892190 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892050 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.892190 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892058 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.892190 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892145 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.892447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892202 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.892447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892277 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.892447 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892322 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.892706 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892683 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.892817 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892725 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.892905 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.892873 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.893221 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.893187 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.893420 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.893394 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.894442 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.894417 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.895365 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.895344 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.897062 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.897039 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.897235 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.897214 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.914354 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.914296 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.916640 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.916571 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.917256 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.917234 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z9dn\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-kube-api-access-5z9dn\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:38.917330 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.917235 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmqv\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-kube-api-access-xnmqv\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:38.980846 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:38.980817 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:39.005695 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.005668 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:39.368793 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.368556 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z"] Oct 29 22:58:39.371445 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:58:39.371385 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe95934_f0ad_421f_b9c1_9c1e6393c5d4.slice/crio-5dfd5d60b2f74f3f6d645ed34b5315af084af58e4ed96c06a36c170f5ec3ac3c WatchSource:0}: Error finding container 5dfd5d60b2f74f3f6d645ed34b5315af084af58e4ed96c06a36c170f5ec3ac3c: Status 404 returned error can't find the container with id 5dfd5d60b2f74f3f6d645ed34b5315af084af58e4ed96c06a36c170f5ec3ac3c Oct 29 22:58:39.381569 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.381526 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:58:39.381693 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.381635 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:58:39.384652 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.384626 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd"] Oct 29 22:58:39.388471 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.387713 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:39.388471 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.387792 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:39.388471 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.387831 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:39.391100 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:58:39.391069 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44af3d7c_5be4_4f8c_a584_14f53dbfef70.slice/crio-2f5b6861d4e61c36d93ec5647d308b10e0659da8d7b5b2d139b259cef362f208 WatchSource:0}: Error finding container 2f5b6861d4e61c36d93ec5647d308b10e0659da8d7b5b2d139b259cef362f208: Status 404 returned error can't find the container with id 2f5b6861d4e61c36d93ec5647d308b10e0659da8d7b5b2d139b259cef362f208 Oct 29 22:58:39.393765 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.393741 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:39.393834 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.393793 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:39.393834 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.393818 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:39.955603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.954943 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" event={"ID":"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4","Type":"ContainerStarted","Data":"cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8"} Oct 29 22:58:39.955603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.955013 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" event={"ID":"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4","Type":"ContainerStarted","Data":"5dfd5d60b2f74f3f6d645ed34b5315af084af58e4ed96c06a36c170f5ec3ac3c"} Oct 29 22:58:39.959976 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.959918 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" event={"ID":"44af3d7c-5be4-4f8c-a584-14f53dbfef70","Type":"ContainerStarted","Data":"6682af431b576bf67a325b810713479738eaf4a1cb50525db42496a12011b2a8"} Oct 29 22:58:39.966723 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.966688 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" event={"ID":"44af3d7c-5be4-4f8c-a584-14f53dbfef70","Type":"ContainerStarted","Data":"2f5b6861d4e61c36d93ec5647d308b10e0659da8d7b5b2d139b259cef362f208"} Oct 29 22:58:39.981645 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:39.981610 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:40.013339 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:40.013305 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:58:40.013966 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:40.013931 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:58:40.014350 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:40.014334 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:40.059287 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:40.059180 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podStartSLOduration=2.059159517 podStartE2EDuration="2.059159517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:40.007107677 +0000 UTC m=+917.123027344" watchObservedRunningTime="2026-10-29 22:58:40.059159517 +0000 UTC m=+917.175079123" Oct 29 22:58:40.065816 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:40.065760 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" podStartSLOduration=2.065740786 podStartE2EDuration="2.065740786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:40.056014756 +0000 UTC m=+917.171934362" watchObservedRunningTime="2026-10-29 22:58:40.065740786 +0000 UTC m=+917.181660385" Oct 29 22:58:40.390606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:40.381876 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:58:40.390606 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:40.381938 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:58:41.007603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:41.007301 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:58:41.007603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:41.007367 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:58:41.014634 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:41.012662 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:58:41.014634 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:41.012720 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 29 22:58:41.437242 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:41.437158 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:58:41.437482 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:41.437450 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:58:41.983993 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:41.983951 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:58:41.984299 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:41.984256 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 29 22:58:42.008972 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.008940 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:58:42.009546 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.009519 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:58:42.050493 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.050452 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r"] Oct 29 22:58:42.063605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.057907 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-031adb66-qvm5r"] Oct 29 22:58:42.107635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.096244 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl"] Oct 29 22:58:42.107635 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.101958 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.109796 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.109764 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl"] Oct 29 22:58:42.126312 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.119765 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:58:42.126312 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.119827 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:58:42.319782 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.319526 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7lh\" (UniqueName: \"kubernetes.io/projected/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-kube-api-access-kw7lh\") pod \"maas-api-revoke-sub-86ed7124-q89wl\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.319782 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.319668 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ed7124-q89wl\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.320029 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.319880 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ed7124-q89wl\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.383601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.382500 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:58:42.383601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.382559 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:58:42.421616 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.420967 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ed7124-q89wl\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.421616 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.421125 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7lh\" (UniqueName: \"kubernetes.io/projected/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-kube-api-access-kw7lh\") pod \"maas-api-revoke-sub-86ed7124-q89wl\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.421616 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.421157 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ed7124-q89wl\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.423603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.422798 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ed7124-q89wl\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.469409 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.469338 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7lh\" (UniqueName: \"kubernetes.io/projected/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-kube-api-access-kw7lh\") pod \"maas-api-revoke-sub-86ed7124-q89wl\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.473604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.469629 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ed7124-q89wl\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.499265 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.498997 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:42.983725 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.983631 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:58:42.983984 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:42.983952 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 29 22:58:43.008007 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:43.007972 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:58:43.008267 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:43.008239 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:58:43.216848 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:43.216797 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30c8d44-74b9-4582-b453-b6b214e6ea93" path="/var/lib/kubelet/pods/b30c8d44-74b9-4582-b453-b6b214e6ea93/volumes" Oct 29 22:58:43.217429 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:43.217408 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl"] Oct 29 22:58:43.390727 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:43.390688 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:58:43.390919 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:43.390756 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 29 22:58:43.990605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:43.990332 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:58:43.990605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:43.990388 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 29 22:58:44.014602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.008614 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:58:44.014602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.008674 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:58:44.251660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.251060 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" event={"ID":"8363fe43-b47f-42d2-9ce2-d7c73ec010a6","Type":"ContainerStarted","Data":"4d61f53add7de6ce26b2b9efe059c5e8e8f8891851d62761289258acbf3f2981"} Oct 29 22:58:44.251660 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.251138 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" event={"ID":"8363fe43-b47f-42d2-9ce2-d7c73ec010a6","Type":"ContainerStarted","Data":"bdc82da4a69ddaeddf9fd0c9e7d3ea99f6219097fa034a33f2942d7425902e69"} Oct 29 22:58:44.311450 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.310885 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" podStartSLOduration=2.310863527 podStartE2EDuration="2.310863527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:44.305828591 +0000 UTC m=+921.421748202" watchObservedRunningTime="2026-10-29 22:58:44.310863527 +0000 UTC m=+921.426783158" Oct 29 22:58:44.714363 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.687913 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:44.714363 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.696977 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:44.727948 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.727918 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 22:58:44.986009 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.985469 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:58:44.986009 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:44.985527 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 29 22:58:45.007868 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:45.006210 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:58:45.007868 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:45.006273 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:58:45.290658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:45.280930 5736 generic.go:356] "Generic (PLEG): container finished" podID="8363fe43-b47f-42d2-9ce2-d7c73ec010a6" containerID="4d61f53add7de6ce26b2b9efe059c5e8e8f8891851d62761289258acbf3f2981" exitCode=0 Oct 29 22:58:45.290658 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:45.282452 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" event={"ID":"8363fe43-b47f-42d2-9ce2-d7c73ec010a6","Type":"ContainerDied","Data":"4d61f53add7de6ce26b2b9efe059c5e8e8f8891851d62761289258acbf3f2981"} Oct 29 22:58:45.982113 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:45.981535 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:58:45.982113 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:45.981631 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 29 22:58:46.005947 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.005912 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:58:46.006111 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.005964 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:58:46.473048 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.473016 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:46.599570 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.599484 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-xks-ca\") pod \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " Oct 29 22:58:46.599757 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.599660 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-openshift-ca\") pod \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " Oct 29 22:58:46.599836 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.599755 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-kube-api-access-kw7lh\" (UniqueName: \"kubernetes.io/projected/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-kube-api-access-kw7lh\") pod \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\" (UID: \"8363fe43-b47f-42d2-9ce2-d7c73ec010a6\") " Oct 29 22:58:46.600071 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.600036 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-openshift-ca" pod "8363fe43-b47f-42d2-9ce2-d7c73ec010a6" (UID: "8363fe43-b47f-42d2-9ce2-d7c73ec010a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:46.600516 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.600493 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:46.601908 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.601879 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-kube-api-access-kw7lh" pod "8363fe43-b47f-42d2-9ce2-d7c73ec010a6" (UID: "8363fe43-b47f-42d2-9ce2-d7c73ec010a6"). InnerVolumeSpecName "kube-api-access-kw7lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:46.602013 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.601989 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-xks-ca" pod "8363fe43-b47f-42d2-9ce2-d7c73ec010a6" (UID: "8363fe43-b47f-42d2-9ce2-d7c73ec010a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:46.701918 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.701891 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw7lh\" (UniqueName: \"kubernetes.io/projected/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-kube-api-access-kw7lh\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:46.701918 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:46.701916 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363fe43-b47f-42d2-9ce2-d7c73ec010a6-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:47.313737 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:47.313651 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" Oct 29 22:58:47.314690 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:47.314663 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl" event={"ID":"8363fe43-b47f-42d2-9ce2-d7c73ec010a6","Type":"ContainerDied","Data":"bdc82da4a69ddaeddf9fd0c9e7d3ea99f6219097fa034a33f2942d7425902e69"} Oct 29 22:58:47.314875 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:47.314859 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc82da4a69ddaeddf9fd0c9e7d3ea99f6219097fa034a33f2942d7425902e69" Oct 29 22:58:47.448782 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:47.448747 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:47.449647 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:47.449620 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:47.450265 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:47.450243 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 22:58:47.511375 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:47.511341 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:47.511988 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:47.511962 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:47.513150 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:47.513127 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 22:58:53.926780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.926741 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w"] Oct 29 22:58:53.927615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.927554 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8363fe43-b47f-42d2-9ce2-d7c73ec010a6" containerName="revoke-keys" Oct 29 22:58:53.927615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.927594 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8363fe43-b47f-42d2-9ce2-d7c73ec010a6" containerName="revoke-keys" Oct 29 22:58:53.927792 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.927748 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8363fe43-b47f-42d2-9ce2-d7c73ec010a6" containerName="revoke-keys" Oct 29 22:58:53.931698 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.931677 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:53.956336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.956309 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:58:53.956336 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.956322 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:58:53.961284 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.961253 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w"] Oct 29 22:58:53.962135 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.962112 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53592152-tzl4w\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:53.962217 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.962177 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b826g\" (UniqueName: \"kubernetes.io/projected/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-kube-api-access-b826g\") pod \"maas-api-revoke-sub-53592152-tzl4w\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:53.962301 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:53.962288 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53592152-tzl4w\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:54.063266 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:54.063235 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b826g\" (UniqueName: \"kubernetes.io/projected/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-kube-api-access-b826g\") pod \"maas-api-revoke-sub-53592152-tzl4w\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:54.063500 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:54.063395 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53592152-tzl4w\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:54.063659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:54.063639 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53592152-tzl4w\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:54.064248 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:54.064224 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53592152-tzl4w\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:54.065767 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:54.065747 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53592152-tzl4w\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:54.084457 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:54.084437 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b826g\" (UniqueName: \"kubernetes.io/projected/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-kube-api-access-b826g\") pod \"maas-api-revoke-sub-53592152-tzl4w\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:54.246186 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:54.246159 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:54.550899 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:54.550871 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w"] Oct 29 22:58:54.552570 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:58:54.552542 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6660ee6a_5ece_4ddf_9e35_50823d1e25ce.slice/crio-351bababeae8274e6d7ddd38548899630a1dc4adc8d751c5b268367a273930fa WatchSource:0}: Error finding container 351bababeae8274e6d7ddd38548899630a1dc4adc8d751c5b268367a273930fa: Status 404 returned error can't find the container with id 351bababeae8274e6d7ddd38548899630a1dc4adc8d751c5b268367a273930fa Oct 29 22:58:55.028798 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.028768 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm"] Oct 29 22:58:55.038477 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.038451 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad74f549-96dsm"] Oct 29 22:58:55.390723 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.390659 5736 generic.go:356] "Generic (PLEG): container finished" podID="6660ee6a-5ece-4ddf-9e35-50823d1e25ce" containerID="f1abc4325abbdc64447622392e4ef84e9c479b0b2b2f3337afd1fd668156654e" exitCode=0 Oct 29 22:58:55.390861 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.390699 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" event={"ID":"6660ee6a-5ece-4ddf-9e35-50823d1e25ce","Type":"ContainerDied","Data":"f1abc4325abbdc64447622392e4ef84e9c479b0b2b2f3337afd1fd668156654e"} Oct 29 22:58:55.391255 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.391229 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" event={"ID":"6660ee6a-5ece-4ddf-9e35-50823d1e25ce","Type":"ContainerStarted","Data":"351bababeae8274e6d7ddd38548899630a1dc4adc8d751c5b268367a273930fa"} Oct 29 22:58:55.712517 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.712484 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7"] Oct 29 22:58:55.717090 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.717069 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:55.739358 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.739288 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7"] Oct 29 22:58:55.790962 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.790928 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b259700-t6rq7\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:55.791113 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.790987 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b259700-t6rq7\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:55.791113 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.791096 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvgx\" (UniqueName: \"kubernetes.io/projected/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-kube-api-access-fcvgx\") pod \"maas-api-revoke-sub-5b259700-t6rq7\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:55.892308 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.892275 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b259700-t6rq7\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:55.892499 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.892324 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b259700-t6rq7\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:55.892499 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.892398 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvgx\" (UniqueName: \"kubernetes.io/projected/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-kube-api-access-fcvgx\") pod \"maas-api-revoke-sub-5b259700-t6rq7\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:55.893057 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.893035 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b259700-t6rq7\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:55.895076 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.895041 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b259700-t6rq7\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:55.914381 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:55.914353 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvgx\" (UniqueName: \"kubernetes.io/projected/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-kube-api-access-fcvgx\") pod \"maas-api-revoke-sub-5b259700-t6rq7\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:56.032940 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.032870 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:56.340680 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.340654 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7"] Oct 29 22:58:56.342330 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:58:56.342296 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21843ebd_cef9_4fd1_9d4e_2d06b1b32a54.slice/crio-15f1d4e8d6fbce1e88787adffc8b3a70672614c6c38057345c4f2ed2e91f5d3b WatchSource:0}: Error finding container 15f1d4e8d6fbce1e88787adffc8b3a70672614c6c38057345c4f2ed2e91f5d3b: Status 404 returned error can't find the container with id 15f1d4e8d6fbce1e88787adffc8b3a70672614c6c38057345c4f2ed2e91f5d3b Oct 29 22:58:56.399636 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.399606 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" event={"ID":"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54","Type":"ContainerStarted","Data":"15f1d4e8d6fbce1e88787adffc8b3a70672614c6c38057345c4f2ed2e91f5d3b"} Oct 29 22:58:56.564901 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.564875 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:56.603264 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.603205 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-openshift-ca\") pod \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " Oct 29 22:58:56.603378 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.603346 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-xks-ca\") pod \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " Oct 29 22:58:56.603433 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.603405 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-kube-api-access-b826g\" (UniqueName: \"kubernetes.io/projected/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-kube-api-access-b826g\") pod \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\" (UID: \"6660ee6a-5ece-4ddf-9e35-50823d1e25ce\") " Oct 29 22:58:56.603525 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.603504 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-openshift-ca" pod "6660ee6a-5ece-4ddf-9e35-50823d1e25ce" (UID: "6660ee6a-5ece-4ddf-9e35-50823d1e25ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:56.604079 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.604058 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:56.605756 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.605715 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-xks-ca" pod "6660ee6a-5ece-4ddf-9e35-50823d1e25ce" (UID: "6660ee6a-5ece-4ddf-9e35-50823d1e25ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:56.605954 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.605928 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-kube-api-access-b826g" pod "6660ee6a-5ece-4ddf-9e35-50823d1e25ce" (UID: "6660ee6a-5ece-4ddf-9e35-50823d1e25ce"). InnerVolumeSpecName "kube-api-access-b826g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:56.705604 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.705552 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b826g\" (UniqueName: \"kubernetes.io/projected/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-kube-api-access-b826g\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:56.705727 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:56.705608 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6660ee6a-5ece-4ddf-9e35-50823d1e25ce-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:56.823714 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:58:56.823679 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:58:57.004516 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:57.004484 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591a40e8-e29f-4fcf-a3e5-945ffd2efc5a" path="/var/lib/kubelet/pods/591a40e8-e29f-4fcf-a3e5-945ffd2efc5a/volumes" Oct 29 22:58:57.408262 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:57.408233 5736 generic.go:356] "Generic (PLEG): container finished" podID="21843ebd-cef9-4fd1-9d4e-2d06b1b32a54" containerID="24ff1b883efd4a24c9268be34423275771e44cd3a5f0271b9a4d71717fdce8f1" exitCode=0 Oct 29 22:58:57.408685 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:57.408297 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" event={"ID":"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54","Type":"ContainerDied","Data":"24ff1b883efd4a24c9268be34423275771e44cd3a5f0271b9a4d71717fdce8f1"} Oct 29 22:58:57.409893 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:57.409875 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" Oct 29 22:58:57.410016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:57.409876 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w" event={"ID":"6660ee6a-5ece-4ddf-9e35-50823d1e25ce","Type":"ContainerDied","Data":"351bababeae8274e6d7ddd38548899630a1dc4adc8d751c5b268367a273930fa"} Oct 29 22:58:57.410016 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:57.409992 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="351bababeae8274e6d7ddd38548899630a1dc4adc8d751c5b268367a273930fa" Oct 29 22:58:58.595221 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.595193 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:58:58.631442 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.631408 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-kube-api-access-fcvgx\" (UniqueName: \"kubernetes.io/projected/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-kube-api-access-fcvgx\") pod \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " Oct 29 22:58:58.631607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.631454 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-openshift-ca\") pod \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " Oct 29 22:58:58.631607 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.631519 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-xks-ca\") pod \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\" (UID: \"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54\") " Oct 29 22:58:58.631967 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.631934 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-openshift-ca" pod "21843ebd-cef9-4fd1-9d4e-2d06b1b32a54" (UID: "21843ebd-cef9-4fd1-9d4e-2d06b1b32a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:58.632355 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.632328 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.640671 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.637164 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-xks-ca" pod "21843ebd-cef9-4fd1-9d4e-2d06b1b32a54" (UID: "21843ebd-cef9-4fd1-9d4e-2d06b1b32a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:58:58.640671 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.637550 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-kube-api-access-fcvgx" pod "21843ebd-cef9-4fd1-9d4e-2d06b1b32a54" (UID: "21843ebd-cef9-4fd1-9d4e-2d06b1b32a54"). InnerVolumeSpecName "kube-api-access-fcvgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:58.733474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.733444 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.733474 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:58.733472 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcvgx\" (UniqueName: \"kubernetes.io/projected/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54-kube-api-access-fcvgx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:58:59.096825 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:58:59.096637 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21843ebd_cef9_4fd1_9d4e_2d06b1b32a54.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:58:59.448017 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:59.432136 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" event={"ID":"21843ebd-cef9-4fd1-9d4e-2d06b1b32a54","Type":"ContainerDied","Data":"15f1d4e8d6fbce1e88787adffc8b3a70672614c6c38057345c4f2ed2e91f5d3b"} Oct 29 22:58:59.448017 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:59.432203 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f1d4e8d6fbce1e88787adffc8b3a70672614c6c38057345c4f2ed2e91f5d3b" Oct 29 22:58:59.448017 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:58:59.432280 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7" Oct 29 22:59:03.031875 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:03.031840 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl"] Oct 29 22:59:03.035774 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:03.035749 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a88eb12-s8dtl"] Oct 29 22:59:05.004340 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:05.004306 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70e218d-0c77-495c-8235-1eb634ea9739" path="/var/lib/kubelet/pods/a70e218d-0c77-495c-8235-1eb634ea9739/volumes" Oct 29 22:59:06.086170 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.086136 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc"] Oct 29 22:59:06.086824 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.086807 5736 cpu_manager.go:414] "containerMap: removing container" podUID="21843ebd-cef9-4fd1-9d4e-2d06b1b32a54" containerName="revoke-keys" Oct 29 22:59:06.086882 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.086826 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21843ebd-cef9-4fd1-9d4e-2d06b1b32a54" containerName="revoke-keys" Oct 29 22:59:06.086882 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.086855 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6660ee6a-5ece-4ddf-9e35-50823d1e25ce" containerName="revoke-keys" Oct 29 22:59:06.086882 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.086861 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6660ee6a-5ece-4ddf-9e35-50823d1e25ce" containerName="revoke-keys" Oct 29 22:59:06.086981 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.086950 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="21843ebd-cef9-4fd1-9d4e-2d06b1b32a54" containerName="revoke-keys" Oct 29 22:59:06.086981 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.086962 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6660ee6a-5ece-4ddf-9e35-50823d1e25ce" containerName="revoke-keys" Oct 29 22:59:06.091662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.091631 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.112294 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.112269 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc"] Oct 29 22:59:06.116790 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.116768 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:59:06.116942 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.116817 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:59:06.118064 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.118039 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba8d575-7b2xc\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.118196 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.118181 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba8d575-7b2xc\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.118267 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.118223 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpt4\" (UniqueName: \"kubernetes.io/projected/8ad24946-8dd9-4340-93d9-3abb9ca312bf-kube-api-access-vlpt4\") pod \"maas-api-revoke-sub-7ba8d575-7b2xc\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.220418 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.220374 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba8d575-7b2xc\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.220716 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.220471 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpt4\" (UniqueName: \"kubernetes.io/projected/8ad24946-8dd9-4340-93d9-3abb9ca312bf-kube-api-access-vlpt4\") pod \"maas-api-revoke-sub-7ba8d575-7b2xc\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.220895 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.220743 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba8d575-7b2xc\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.221650 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.221622 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba8d575-7b2xc\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.224093 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.224067 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba8d575-7b2xc\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.242036 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.242011 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpt4\" (UniqueName: \"kubernetes.io/projected/8ad24946-8dd9-4340-93d9-3abb9ca312bf-kube-api-access-vlpt4\") pod \"maas-api-revoke-sub-7ba8d575-7b2xc\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.407748 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.407658 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:06.746902 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:06.746877 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc"] Oct 29 22:59:06.749683 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:59:06.749652 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad24946_8dd9_4340_93d9_3abb9ca312bf.slice/crio-6cd0dc45e3f4064e906d00b8262269245c5b7a9c7866fbfd8bd93ca643465e36 WatchSource:0}: Error finding container 6cd0dc45e3f4064e906d00b8262269245c5b7a9c7866fbfd8bd93ca643465e36: Status 404 returned error can't find the container with id 6cd0dc45e3f4064e906d00b8262269245c5b7a9c7866fbfd8bd93ca643465e36 Oct 29 22:59:07.029764 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:07.029688 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt"] Oct 29 22:59:07.039566 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:07.039539 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c90888-m7jvt"] Oct 29 22:59:07.529415 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:07.529373 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" event={"ID":"8ad24946-8dd9-4340-93d9-3abb9ca312bf","Type":"ContainerStarted","Data":"05c9596e065ad00d3bf1e20310170b9ab07df844cf2a07ef15ceb2733759f8d1"} Oct 29 22:59:07.529915 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:07.529433 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" event={"ID":"8ad24946-8dd9-4340-93d9-3abb9ca312bf","Type":"ContainerStarted","Data":"6cd0dc45e3f4064e906d00b8262269245c5b7a9c7866fbfd8bd93ca643465e36"} Oct 29 22:59:07.569936 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:07.569887 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" podStartSLOduration=1.569871137 podStartE2EDuration="1.569871137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:07.565814822 +0000 UTC m=+944.681734429" watchObservedRunningTime="2026-10-29 22:59:07.569871137 +0000 UTC m=+944.685790765" Oct 29 22:59:08.538215 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:08.538177 5736 generic.go:356] "Generic (PLEG): container finished" podID="8ad24946-8dd9-4340-93d9-3abb9ca312bf" containerID="05c9596e065ad00d3bf1e20310170b9ab07df844cf2a07ef15ceb2733759f8d1" exitCode=0 Oct 29 22:59:08.538631 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:08.538226 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" event={"ID":"8ad24946-8dd9-4340-93d9-3abb9ca312bf","Type":"ContainerDied","Data":"05c9596e065ad00d3bf1e20310170b9ab07df844cf2a07ef15ceb2733759f8d1"} Oct 29 22:59:09.004776 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.004744 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd25c55-e25f-41a2-be9c-96ee70cf06da" path="/var/lib/kubelet/pods/acd25c55-e25f-41a2-be9c-96ee70cf06da/volumes" Oct 29 22:59:09.694780 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.694740 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:09.773376 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.773347 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-openshift-ca\") pod \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " Oct 29 22:59:09.773524 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.773463 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-xks-ca\") pod \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " Oct 29 22:59:09.773712 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.773693 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad24946-8dd9-4340-93d9-3abb9ca312bf-kube-api-access-vlpt4\" (UniqueName: \"kubernetes.io/projected/8ad24946-8dd9-4340-93d9-3abb9ca312bf-kube-api-access-vlpt4\") pod \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\" (UID: \"8ad24946-8dd9-4340-93d9-3abb9ca312bf\") " Oct 29 22:59:09.773893 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.773862 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-openshift-ca" pod "8ad24946-8dd9-4340-93d9-3abb9ca312bf" (UID: "8ad24946-8dd9-4340-93d9-3abb9ca312bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:09.774701 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.774679 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:09.776333 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.776307 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad24946-8dd9-4340-93d9-3abb9ca312bf-kube-api-access-vlpt4" pod "8ad24946-8dd9-4340-93d9-3abb9ca312bf" (UID: "8ad24946-8dd9-4340-93d9-3abb9ca312bf"). InnerVolumeSpecName "kube-api-access-vlpt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:09.779085 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.779059 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-xks-ca" pod "8ad24946-8dd9-4340-93d9-3abb9ca312bf" (UID: "8ad24946-8dd9-4340-93d9-3abb9ca312bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:09.895666 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.879028 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad24946-8dd9-4340-93d9-3abb9ca312bf-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:09.895666 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:09.879069 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlpt4\" (UniqueName: \"kubernetes.io/projected/8ad24946-8dd9-4340-93d9-3abb9ca312bf-kube-api-access-vlpt4\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:10.553608 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:10.553550 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" Oct 29 22:59:10.553793 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:10.553622 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc" event={"ID":"8ad24946-8dd9-4340-93d9-3abb9ca312bf","Type":"ContainerDied","Data":"6cd0dc45e3f4064e906d00b8262269245c5b7a9c7866fbfd8bd93ca643465e36"} Oct 29 22:59:10.553793 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:10.553675 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cd0dc45e3f4064e906d00b8262269245c5b7a9c7866fbfd8bd93ca643465e36" Oct 29 22:59:21.031631 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:21.031594 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm"] Oct 29 22:59:21.040810 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:21.040782 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0681236b-42jqm"] Oct 29 22:59:23.009871 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:23.009834 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3162219-21f7-434f-983b-61d2eeefed4b" path="/var/lib/kubelet/pods/e3162219-21f7-434f-983b-61d2eeefed4b/volumes" Oct 29 22:59:26.824617 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:59:26.824552 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:59:28.405947 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.405916 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7"] Oct 29 22:59:28.406530 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.406515 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8ad24946-8dd9-4340-93d9-3abb9ca312bf" containerName="revoke-keys" Oct 29 22:59:28.406609 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.406532 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad24946-8dd9-4340-93d9-3abb9ca312bf" containerName="revoke-keys" Oct 29 22:59:28.406683 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.406673 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad24946-8dd9-4340-93d9-3abb9ca312bf" containerName="revoke-keys" Oct 29 22:59:28.411897 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.411876 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.435296 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.435267 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:59:28.435662 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.435627 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:59:28.441390 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.441354 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7"] Oct 29 22:59:28.507686 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.507658 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ptp\" (UniqueName: \"kubernetes.io/projected/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-kube-api-access-b4ptp\") pod \"maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.507855 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.507840 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.507924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.507912 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.609050 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.609018 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ptp\" (UniqueName: \"kubernetes.io/projected/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-kube-api-access-b4ptp\") pod \"maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.609221 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.609195 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.609277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.609265 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.609924 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.609905 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.611535 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.611508 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.630498 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.630467 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ptp\" (UniqueName: \"kubernetes.io/projected/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-kube-api-access-b4ptp\") pod \"maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:28.727249 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:28.727221 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:29.033081 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:29.033056 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7"] Oct 29 22:59:29.035476 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:59:29.035447 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6ce08e_6d1a_435e_b97a_80bfcd3b5463.slice/crio-da21f6f44f1465a823b8f4ce7db4496dbfcf4a7c1f3e716665d916e8aa1ac9e1 WatchSource:0}: Error finding container da21f6f44f1465a823b8f4ce7db4496dbfcf4a7c1f3e716665d916e8aa1ac9e1: Status 404 returned error can't find the container with id da21f6f44f1465a823b8f4ce7db4496dbfcf4a7c1f3e716665d916e8aa1ac9e1 Oct 29 22:59:29.731231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:29.731166 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" event={"ID":"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463","Type":"ContainerStarted","Data":"03efba9318eac56d593c06d808f7ce258c03f9dd545a43fed0c1b326f04d052c"} Oct 29 22:59:29.731801 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:29.731302 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" event={"ID":"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463","Type":"ContainerStarted","Data":"da21f6f44f1465a823b8f4ce7db4496dbfcf4a7c1f3e716665d916e8aa1ac9e1"} Oct 29 22:59:29.773250 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:29.773188 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" podStartSLOduration=1.7731705880000002 podStartE2EDuration="1.773170588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:29.769503419 +0000 UTC m=+966.885423024" watchObservedRunningTime="2026-10-29 22:59:29.773170588 +0000 UTC m=+966.889090195" Oct 29 22:59:30.740261 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:30.740225 5736 generic.go:356] "Generic (PLEG): container finished" podID="3b6ce08e-6d1a-435e-b97a-80bfcd3b5463" containerID="03efba9318eac56d593c06d808f7ce258c03f9dd545a43fed0c1b326f04d052c" exitCode=0 Oct 29 22:59:30.740679 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:30.740308 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" event={"ID":"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463","Type":"ContainerDied","Data":"03efba9318eac56d593c06d808f7ce258c03f9dd545a43fed0c1b326f04d052c"} Oct 29 22:59:30.951600 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:30.950055 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq"] Oct 29 22:59:30.954844 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:30.954814 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.009533 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.009402 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq"] Oct 29 22:59:31.038543 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.038512 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8"] Oct 29 22:59:31.044297 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.044271 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d8f66-469p8"] Oct 29 22:59:31.047309 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.047281 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.047520 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.047496 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.047637 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.047556 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.047756 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.047739 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.047839 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.047824 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.047945 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.047932 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.048007 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.047974 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.048053 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.047999 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5n52\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-kube-api-access-m5n52\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.048133 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.048116 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.161621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.155504 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.161621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.155569 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.161621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.155607 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5n52\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-kube-api-access-m5n52\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.161621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.155650 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.161621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.155737 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.161621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.155883 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.161621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.155930 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.161621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.156007 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.161621 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.156095 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.164155 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.164119 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.164985 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.164775 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.165750 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.165724 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.166993 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.166257 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.188601 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.185707 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.208619 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.206224 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.208619 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.206868 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.223977 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.223138 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5n52\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-kube-api-access-m5n52\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.224356 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.224324 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.276688 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.276589 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:31.608026 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.608001 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq"] Oct 29 22:59:31.609552 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:59:31.609511 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e48f4af_7aa3_41f1_9d6d_06aa2fee3108.slice/crio-24756f629f8fec15d8a65ed160a1b625cdb600debed90da956bab03fd7a8a7fb WatchSource:0}: Error finding container 24756f629f8fec15d8a65ed160a1b625cdb600debed90da956bab03fd7a8a7fb: Status 404 returned error can't find the container with id 24756f629f8fec15d8a65ed160a1b625cdb600debed90da956bab03fd7a8a7fb Oct 29 22:59:31.612171 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.612137 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:59:31.612278 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.612216 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:59:31.612278 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.612259 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:59:31.750442 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.750392 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" event={"ID":"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108","Type":"ContainerStarted","Data":"09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa"} Oct 29 22:59:31.750923 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.750461 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" event={"ID":"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108","Type":"ContainerStarted","Data":"24756f629f8fec15d8a65ed160a1b625cdb600debed90da956bab03fd7a8a7fb"} Oct 29 22:59:31.800633 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.800088 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podStartSLOduration=1.800067384 podStartE2EDuration="1.800067384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:31.799371135 +0000 UTC m=+968.915290740" watchObservedRunningTime="2026-10-29 22:59:31.800067384 +0000 UTC m=+968.915986990" Oct 29 22:59:31.910025 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:31.909997 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:32.071677 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.071633 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-kube-api-access-b4ptp\" (UniqueName: \"kubernetes.io/projected/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-kube-api-access-b4ptp\") pod \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " Oct 29 22:59:32.071824 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.071778 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-openshift-ca\") pod \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " Oct 29 22:59:32.071891 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.071878 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-xks-ca\") pod \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\" (UID: \"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463\") " Oct 29 22:59:32.072158 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.072116 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-openshift-ca" pod "3b6ce08e-6d1a-435e-b97a-80bfcd3b5463" (UID: "3b6ce08e-6d1a-435e-b97a-80bfcd3b5463"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:32.072509 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.072490 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:32.073827 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.073801 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-kube-api-access-b4ptp" pod "3b6ce08e-6d1a-435e-b97a-80bfcd3b5463" (UID: "3b6ce08e-6d1a-435e-b97a-80bfcd3b5463"). InnerVolumeSpecName "kube-api-access-b4ptp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:32.073917 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.073904 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-xks-ca" pod "3b6ce08e-6d1a-435e-b97a-80bfcd3b5463" (UID: "3b6ce08e-6d1a-435e-b97a-80bfcd3b5463"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:32.174609 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.174490 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:32.174772 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.174645 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ptp\" (UniqueName: \"kubernetes.io/projected/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463-kube-api-access-b4ptp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:32.281250 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.281219 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:32.284755 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.283689 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:59:32.284755 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.283754 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 29 22:59:32.764179 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.764144 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" Oct 29 22:59:32.764570 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.764179 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7" event={"ID":"3b6ce08e-6d1a-435e-b97a-80bfcd3b5463","Type":"ContainerDied","Data":"da21f6f44f1465a823b8f4ce7db4496dbfcf4a7c1f3e716665d916e8aa1ac9e1"} Oct 29 22:59:32.764570 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:32.764257 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da21f6f44f1465a823b8f4ce7db4496dbfcf4a7c1f3e716665d916e8aa1ac9e1" Oct 29 22:59:33.073754 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:33.062728 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202674fe-a5f9-45cf-9641-84882f7ebe76" path="/var/lib/kubelet/pods/202674fe-a5f9-45cf-9641-84882f7ebe76/volumes" Oct 29 22:59:33.290605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:33.284037 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:59:33.290605 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:33.284098 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 29 22:59:34.277722 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:34.277674 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:59:34.278245 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:34.277735 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 29 22:59:35.282611 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:35.281914 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:59:35.282611 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:35.281975 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 29 22:59:36.014602 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.010706 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:36.228816 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.228750 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:36.245744 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.238444 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 29 22:59:36.396656 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.387719 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:59:36.421832 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.421785 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 29 22:59:36.425813 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.425699 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 22:59:36.478608 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.441549 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 29 22:59:36.479145 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.479099 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:36.500829 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.500783 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 22:59:36.501366 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.501331 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 22:59:36.501854 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.501833 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 22:59:36.506499 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:36.485753 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:37.277646 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:37.277611 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:59:37.277807 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:37.277669 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 29 22:59:38.277502 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:38.277467 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:59:38.277939 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:38.277561 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 29 22:59:39.833984 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.833926 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp"] Oct 29 22:59:39.836235 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.836206 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3b6ce08e-6d1a-435e-b97a-80bfcd3b5463" containerName="revoke-keys" Oct 29 22:59:39.836235 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.836233 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6ce08e-6d1a-435e-b97a-80bfcd3b5463" containerName="revoke-keys" Oct 29 22:59:39.836603 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.836572 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6ce08e-6d1a-435e-b97a-80bfcd3b5463" containerName="revoke-keys" Oct 29 22:59:39.870159 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.870042 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp"] Oct 29 22:59:39.870159 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.870162 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:39.891028 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.890991 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:59:39.913680 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.907127 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:59:39.995744 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.995705 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wvwz\" (UniqueName: \"kubernetes.io/projected/94532a44-334e-48db-ad04-dd0ee1ac230f-kube-api-access-4wvwz\") pod \"maas-api-revoke-sub-82614eca-7bllp\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:39.995938 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.995772 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82614eca-7bllp\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:39.996015 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:39.995982 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82614eca-7bllp\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:40.097668 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.097573 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvwz\" (UniqueName: \"kubernetes.io/projected/94532a44-334e-48db-ad04-dd0ee1ac230f-kube-api-access-4wvwz\") pod \"maas-api-revoke-sub-82614eca-7bllp\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:40.097668 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.097651 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82614eca-7bllp\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:40.097916 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.097850 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82614eca-7bllp\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:40.098417 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.098386 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82614eca-7bllp\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:40.100207 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.100190 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82614eca-7bllp\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:40.118490 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.118461 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvwz\" (UniqueName: \"kubernetes.io/projected/94532a44-334e-48db-ad04-dd0ee1ac230f-kube-api-access-4wvwz\") pod \"maas-api-revoke-sub-82614eca-7bllp\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:40.267353 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.267321 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:40.348938 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.328709 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:59:40.348938 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.328802 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:59:40.656228 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.653037 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp"] Oct 29 22:59:40.657648 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:59:40.657615 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94532a44_334e_48db_ad04_dd0ee1ac230f.slice/crio-600c24da0b2dadc2f85cc2bfde28e87d6082db189b44c3911c1a8d2bd6ac1637 WatchSource:0}: Error finding container 600c24da0b2dadc2f85cc2bfde28e87d6082db189b44c3911c1a8d2bd6ac1637: Status 404 returned error can't find the container with id 600c24da0b2dadc2f85cc2bfde28e87d6082db189b44c3911c1a8d2bd6ac1637 Oct 29 22:59:40.901423 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.901388 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:40.904549 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.904521 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:40.934173 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.934085 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 22:59:40.954812 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.954775 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-a7914d"] Oct 29 22:59:40.962053 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.961552 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:40.964106 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.964056 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-a7914d"] Oct 29 22:59:40.976640 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:40.976611 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xdwj4\"" Oct 29 22:59:41.031754 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:41.031720 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mswjh\" (UniqueName: \"kubernetes.io/projected/1348501a-589d-4577-9b79-4e4e6b6f0b92-kube-api-access-mswjh\") pod \"test-curl-7896-a7914d\" (UID: \"1348501a-589d-4577-9b79-4e4e6b6f0b92\") " pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:41.135005 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:41.134964 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mswjh\" (UniqueName: \"kubernetes.io/projected/1348501a-589d-4577-9b79-4e4e6b6f0b92-kube-api-access-mswjh\") pod \"test-curl-7896-a7914d\" (UID: \"1348501a-589d-4577-9b79-4e4e6b6f0b92\") " pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:41.156687 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:41.156656 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mswjh\" (UniqueName: \"kubernetes.io/projected/1348501a-589d-4577-9b79-4e4e6b6f0b92-kube-api-access-mswjh\") pod \"test-curl-7896-a7914d\" (UID: \"1348501a-589d-4577-9b79-4e4e6b6f0b92\") " pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:41.318574 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:41.318540 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:41.608685 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:41.607633 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" event={"ID":"94532a44-334e-48db-ad04-dd0ee1ac230f","Type":"ContainerStarted","Data":"9589d963e9a02de71446c51359a9a0d881673c496e0f4b0c9abd3a9bff9d98f7"} Oct 29 22:59:41.608685 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:41.607699 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" event={"ID":"94532a44-334e-48db-ad04-dd0ee1ac230f","Type":"ContainerStarted","Data":"600c24da0b2dadc2f85cc2bfde28e87d6082db189b44c3911c1a8d2bd6ac1637"} Oct 29 22:59:41.629035 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:41.629007 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-a7914d"] Oct 29 22:59:41.631247 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:59:41.631219 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1348501a_589d_4577_9b79_4e4e6b6f0b92.slice/crio-7b7fa6474b86067b57c4e8663da7e30f4d43597f375e686b423db6670c2cc68e WatchSource:0}: Error finding container 7b7fa6474b86067b57c4e8663da7e30f4d43597f375e686b423db6670c2cc68e: Status 404 returned error can't find the container with id 7b7fa6474b86067b57c4e8663da7e30f4d43597f375e686b423db6670c2cc68e Oct 29 22:59:41.634118 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:41.634090 5736 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:59:41.650380 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:41.650319 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" podStartSLOduration=2.6502976350000003 podStartE2EDuration="2.650297635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:41.643218577 +0000 UTC m=+978.759138183" watchObservedRunningTime="2026-10-29 22:59:41.650297635 +0000 UTC m=+978.766217267" Oct 29 22:59:42.620095 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:42.620052 5736 generic.go:356] "Generic (PLEG): container finished" podID="94532a44-334e-48db-ad04-dd0ee1ac230f" containerID="9589d963e9a02de71446c51359a9a0d881673c496e0f4b0c9abd3a9bff9d98f7" exitCode=0 Oct 29 22:59:42.620533 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:42.620147 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" event={"ID":"94532a44-334e-48db-ad04-dd0ee1ac230f","Type":"ContainerDied","Data":"9589d963e9a02de71446c51359a9a0d881673c496e0f4b0c9abd3a9bff9d98f7"} Oct 29 22:59:42.622384 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:42.622356 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-a7914d" event={"ID":"1348501a-589d-4577-9b79-4e4e6b6f0b92","Type":"ContainerStarted","Data":"7b7fa6474b86067b57c4e8663da7e30f4d43597f375e686b423db6670c2cc68e"} Oct 29 22:59:43.824896 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:43.824869 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:43.914642 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:43.913148 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-openshift-ca\") pod \"94532a44-334e-48db-ad04-dd0ee1ac230f\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " Oct 29 22:59:43.914642 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:43.913201 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94532a44-334e-48db-ad04-dd0ee1ac230f-kube-api-access-4wvwz\" (UniqueName: \"kubernetes.io/projected/94532a44-334e-48db-ad04-dd0ee1ac230f-kube-api-access-4wvwz\") pod \"94532a44-334e-48db-ad04-dd0ee1ac230f\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " Oct 29 22:59:43.914642 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:43.913446 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-xks-ca\") pod \"94532a44-334e-48db-ad04-dd0ee1ac230f\" (UID: \"94532a44-334e-48db-ad04-dd0ee1ac230f\") " Oct 29 22:59:43.920722 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:43.916269 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-openshift-ca" pod "94532a44-334e-48db-ad04-dd0ee1ac230f" (UID: "94532a44-334e-48db-ad04-dd0ee1ac230f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:43.922268 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:43.922215 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94532a44-334e-48db-ad04-dd0ee1ac230f-kube-api-access-4wvwz" pod "94532a44-334e-48db-ad04-dd0ee1ac230f" (UID: "94532a44-334e-48db-ad04-dd0ee1ac230f"). InnerVolumeSpecName "kube-api-access-4wvwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:43.924071 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:43.924049 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-xks-ca" pod "94532a44-334e-48db-ad04-dd0ee1ac230f" (UID: "94532a44-334e-48db-ad04-dd0ee1ac230f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:44.016027 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:44.015987 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:44.016027 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:44.016028 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wvwz\" (UniqueName: \"kubernetes.io/projected/94532a44-334e-48db-ad04-dd0ee1ac230f-kube-api-access-4wvwz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:44.016247 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:44.016046 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94532a44-334e-48db-ad04-dd0ee1ac230f-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:44.642659 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:44.642613 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" event={"ID":"94532a44-334e-48db-ad04-dd0ee1ac230f","Type":"ContainerDied","Data":"600c24da0b2dadc2f85cc2bfde28e87d6082db189b44c3911c1a8d2bd6ac1637"} Oct 29 22:59:44.642845 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:44.642687 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="600c24da0b2dadc2f85cc2bfde28e87d6082db189b44c3911c1a8d2bd6ac1637" Oct 29 22:59:44.642845 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:44.642630 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp" Oct 29 22:59:44.644631 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:44.644598 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-a7914d" event={"ID":"1348501a-589d-4577-9b79-4e4e6b6f0b92","Type":"ContainerStarted","Data":"63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d"} Oct 29 22:59:44.685528 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:44.685477 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7896-a7914d" podStartSLOduration=1.865227668 podStartE2EDuration="4.685461813s" totalImagesPullingTime="2.820234145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:44.680799449 +0000 UTC m=+981.796719054" watchObservedRunningTime="2026-10-29 22:59:44.685461813 +0000 UTC m=+981.801381419" Oct 29 22:59:45.620173 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:45.620135 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-a7914d"] Oct 29 22:59:45.625133 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:45.625095 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-a7914d"] Oct 29 22:59:46.266592 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.266538 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w"] Oct 29 22:59:46.267231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.267208 5736 cpu_manager.go:414] "containerMap: removing container" podUID="94532a44-334e-48db-ad04-dd0ee1ac230f" containerName="revoke-keys" Oct 29 22:59:46.267231 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.267231 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94532a44-334e-48db-ad04-dd0ee1ac230f" containerName="revoke-keys" Oct 29 22:59:46.267406 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.267287 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1348501a-589d-4577-9b79-4e4e6b6f0b92" containerName="test-curl-7896-a7914d" Oct 29 22:59:46.267406 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.267295 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1348501a-589d-4577-9b79-4e4e6b6f0b92" containerName="test-curl-7896-a7914d" Oct 29 22:59:46.267509 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.267457 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1348501a-589d-4577-9b79-4e4e6b6f0b92" containerName="test-curl-7896-a7914d" Oct 29 22:59:46.267509 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.267480 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="94532a44-334e-48db-ad04-dd0ee1ac230f" containerName="revoke-keys" Oct 29 22:59:46.271352 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.271329 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.285180 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.285160 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:59:46.295470 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.295446 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 22:59:46.301115 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.301085 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w"] Oct 29 22:59:46.455905 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.455873 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-nm59w\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.456104 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.456087 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7md9v\" (UniqueName: \"kubernetes.io/projected/5848ab58-08cc-4029-8202-c81f5c9fb186-kube-api-access-7md9v\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-nm59w\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.456201 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.456186 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-nm59w\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.558117 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.558052 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7md9v\" (UniqueName: \"kubernetes.io/projected/5848ab58-08cc-4029-8202-c81f5c9fb186-kube-api-access-7md9v\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-nm59w\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.558117 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.558113 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-nm59w\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.558292 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.558171 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-nm59w\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.558841 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.558815 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-nm59w\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.560440 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.560424 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-nm59w\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.579493 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.579471 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7md9v\" (UniqueName: \"kubernetes.io/projected/5848ab58-08cc-4029-8202-c81f5c9fb186-kube-api-access-7md9v\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-nm59w\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.585289 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.585266 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:46.670514 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.670449 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7896-a7914d" podUID="1348501a-589d-4577-9b79-4e4e6b6f0b92" containerName="test-curl-7896-a7914d" containerID="cri-o://63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d" gracePeriod=2 Oct 29 22:59:46.682323 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:59:46.682281 5736 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-a7914d\" is forbidden: User \"system:node:ip-10-0-143-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-172.ec2.internal' and this object" podUID="1348501a-589d-4577-9b79-4e4e6b6f0b92" pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:46.895499 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:46.895470 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w"] Oct 29 22:59:46.896282 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:59:46.896254 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5848ab58_08cc_4029_8202_c81f5c9fb186.slice/crio-affcaeb410e9e338b3a090763cf9f3ea641cbae9ac9c1676c1a771351bae0168 WatchSource:0}: Error finding container affcaeb410e9e338b3a090763cf9f3ea641cbae9ac9c1676c1a771351bae0168: Status 404 returned error can't find the container with id affcaeb410e9e338b3a090763cf9f3ea641cbae9ac9c1676c1a771351bae0168 Oct 29 22:59:47.679212 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:47.679136 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" event={"ID":"5848ab58-08cc-4029-8202-c81f5c9fb186","Type":"ContainerStarted","Data":"5cdf7883b19798b822dffdff191da89f5dac473ef5a665d6d5c0194be7e6da3e"} Oct 29 22:59:47.679212 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:47.679183 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" event={"ID":"5848ab58-08cc-4029-8202-c81f5c9fb186","Type":"ContainerStarted","Data":"affcaeb410e9e338b3a090763cf9f3ea641cbae9ac9c1676c1a771351bae0168"} Oct 29 22:59:47.720428 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:47.720378 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" podStartSLOduration=1.720364556 podStartE2EDuration="1.720364556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:47.715450036 +0000 UTC m=+984.831369676" watchObservedRunningTime="2026-10-29 22:59:47.720364556 +0000 UTC m=+984.836284201" Oct 29 22:59:48.689970 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:48.689939 5736 generic.go:356] "Generic (PLEG): container finished" podID="5848ab58-08cc-4029-8202-c81f5c9fb186" containerID="5cdf7883b19798b822dffdff191da89f5dac473ef5a665d6d5c0194be7e6da3e" exitCode=0 Oct 29 22:59:48.690277 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:48.690023 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" event={"ID":"5848ab58-08cc-4029-8202-c81f5c9fb186","Type":"ContainerDied","Data":"5cdf7883b19798b822dffdff191da89f5dac473ef5a665d6d5c0194be7e6da3e"} Oct 29 22:59:48.830615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:48.820919 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92"] Oct 29 22:59:48.830615 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:48.829156 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 22:59:48.876709 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:48.845835 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-475876\"" Oct 29 22:59:48.876709 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:48.860712 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-475876-dockercfg-lq9jn\"" Oct 29 22:59:48.945094 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:48.941941 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg95d\" (UniqueName: \"kubernetes.io/projected/357963fa-2969-45f8-b032-923cf8ad1003-kube-api-access-gg95d\") pod \"payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92\" (UID: \"357963fa-2969-45f8-b032-923cf8ad1003\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 22:59:48.945094 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:48.942038 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357963fa-2969-45f8-b032-923cf8ad1003-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92\" (UID: \"357963fa-2969-45f8-b032-923cf8ad1003\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 22:59:48.972905 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:48.972151 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92"] Oct 29 22:59:49.048888 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.048839 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg95d\" (UniqueName: \"kubernetes.io/projected/357963fa-2969-45f8-b032-923cf8ad1003-kube-api-access-gg95d\") pod \"payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92\" (UID: \"357963fa-2969-45f8-b032-923cf8ad1003\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 22:59:49.049175 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.049149 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357963fa-2969-45f8-b032-923cf8ad1003-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92\" (UID: \"357963fa-2969-45f8-b032-923cf8ad1003\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 22:59:49.050408 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.050385 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357963fa-2969-45f8-b032-923cf8ad1003-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92\" (UID: \"357963fa-2969-45f8-b032-923cf8ad1003\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 22:59:49.073653 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.073627 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg95d\" (UniqueName: \"kubernetes.io/projected/357963fa-2969-45f8-b032-923cf8ad1003-kube-api-access-gg95d\") pod \"payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92\" (UID: \"357963fa-2969-45f8-b032-923cf8ad1003\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 22:59:49.180046 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.180020 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:49.261252 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.261214 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1348501a-589d-4577-9b79-4e4e6b6f0b92-kube-api-access-mswjh\" (UniqueName: \"kubernetes.io/projected/1348501a-589d-4577-9b79-4e4e6b6f0b92-kube-api-access-mswjh\") pod \"1348501a-589d-4577-9b79-4e4e6b6f0b92\" (UID: \"1348501a-589d-4577-9b79-4e4e6b6f0b92\") " Oct 29 22:59:49.262078 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.262054 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 22:59:49.263567 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.263539 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1348501a-589d-4577-9b79-4e4e6b6f0b92-kube-api-access-mswjh" pod "1348501a-589d-4577-9b79-4e4e6b6f0b92" (UID: "1348501a-589d-4577-9b79-4e4e6b6f0b92"). InnerVolumeSpecName "kube-api-access-mswjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:49.373551 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.373518 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mswjh\" (UniqueName: \"kubernetes.io/projected/1348501a-589d-4577-9b79-4e4e6b6f0b92-kube-api-access-mswjh\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:49.576831 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.576804 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92"] Oct 29 22:59:49.578275 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:59:49.578250 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357963fa_2969_45f8_b032_923cf8ad1003.slice/crio-6902ec1ad15faae2d4a9124cb29a25f72fa9b29df154c0b66f2d86c990ced362 WatchSource:0}: Error finding container 6902ec1ad15faae2d4a9124cb29a25f72fa9b29df154c0b66f2d86c990ced362: Status 404 returned error can't find the container with id 6902ec1ad15faae2d4a9124cb29a25f72fa9b29df154c0b66f2d86c990ced362 Oct 29 22:59:49.699689 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.699662 5736 generic.go:356] "Generic (PLEG): container finished" podID="1348501a-589d-4577-9b79-4e4e6b6f0b92" containerID="63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d" exitCode=137 Oct 29 22:59:49.700068 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.699707 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:49.700068 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.699770 5736 scope.go:122] "RemoveContainer" containerID="63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d" Oct 29 22:59:49.702012 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.701985 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" event={"ID":"357963fa-2969-45f8-b032-923cf8ad1003","Type":"ContainerStarted","Data":"4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b"} Oct 29 22:59:49.702112 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.702032 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" event={"ID":"357963fa-2969-45f8-b032-923cf8ad1003","Type":"ContainerStarted","Data":"6902ec1ad15faae2d4a9124cb29a25f72fa9b29df154c0b66f2d86c990ced362"} Oct 29 22:59:49.711897 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.711878 5736 scope.go:122] "RemoveContainer" containerID="63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d" Oct 29 22:59:49.712220 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:59:49.712195 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d\": container with ID starting with 63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d not found: ID does not exist" containerID="63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d" Oct 29 22:59:49.712284 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.712232 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d"} err="failed to get container status \"63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d\": rpc error: code = NotFound desc = could not find container \"63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d\": container with ID starting with 63988425ed442e7791a7a4271ddf36da1c3e4b35d4141d573d3e798aab51119d not found: ID does not exist" Oct 29 22:59:49.741638 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.741496 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" podStartSLOduration=1.74148052 podStartE2EDuration="1.74148052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:49.737915369 +0000 UTC m=+986.853834974" watchObservedRunningTime="2026-10-29 22:59:49.74148052 +0000 UTC m=+986.857400129" Oct 29 22:59:49.752052 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:59:49.752014 5736 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-a7914d\" is forbidden: User \"system:node:ip-10-0-143-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-172.ec2.internal' and this object" podUID="1348501a-589d-4577-9b79-4e4e6b6f0b92" pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:49.881958 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.881932 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:49.931274 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:59:49.929689 5736 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-a7914d\" is forbidden: User \"system:node:ip-10-0-143-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-172.ec2.internal' and this object" podUID="1348501a-589d-4577-9b79-4e4e6b6f0b92" pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:49.988010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.983404 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-openshift-ca\") pod \"5848ab58-08cc-4029-8202-c81f5c9fb186\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " Oct 29 22:59:49.988010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.983832 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-xks-ca\") pod \"5848ab58-08cc-4029-8202-c81f5c9fb186\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " Oct 29 22:59:49.988010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.983904 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5848ab58-08cc-4029-8202-c81f5c9fb186-kube-api-access-7md9v\" (UniqueName: \"kubernetes.io/projected/5848ab58-08cc-4029-8202-c81f5c9fb186-kube-api-access-7md9v\") pod \"5848ab58-08cc-4029-8202-c81f5c9fb186\" (UID: \"5848ab58-08cc-4029-8202-c81f5c9fb186\") " Oct 29 22:59:49.988010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.983916 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-openshift-ca" pod "5848ab58-08cc-4029-8202-c81f5c9fb186" (UID: "5848ab58-08cc-4029-8202-c81f5c9fb186"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:49.988010 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.987502 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:49.993756 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.993712 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-xks-ca" pod "5848ab58-08cc-4029-8202-c81f5c9fb186" (UID: "5848ab58-08cc-4029-8202-c81f5c9fb186"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:49.993756 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:49.993745 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5848ab58-08cc-4029-8202-c81f5c9fb186-kube-api-access-7md9v" pod "5848ab58-08cc-4029-8202-c81f5c9fb186" (UID: "5848ab58-08cc-4029-8202-c81f5c9fb186"). InnerVolumeSpecName "kube-api-access-7md9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:50.089428 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:50.089339 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848ab58-08cc-4029-8202-c81f5c9fb186-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.089428 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:50.089379 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7md9v\" (UniqueName: \"kubernetes.io/projected/5848ab58-08cc-4029-8202-c81f5c9fb186-kube-api-access-7md9v\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 22:59:50.727709 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:50.727681 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" Oct 29 22:59:50.728080 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:50.727709 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w" event={"ID":"5848ab58-08cc-4029-8202-c81f5c9fb186","Type":"ContainerDied","Data":"affcaeb410e9e338b3a090763cf9f3ea641cbae9ac9c1676c1a771351bae0168"} Oct 29 22:59:50.728080 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:50.727794 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="affcaeb410e9e338b3a090763cf9f3ea641cbae9ac9c1676c1a771351bae0168" Oct 29 22:59:50.728080 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:50.727851 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 22:59:50.761821 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:59:50.761781 5736 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-a7914d\" is forbidden: User \"system:node:ip-10-0-143-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-172.ec2.internal' and this object" podUID="1348501a-589d-4577-9b79-4e4e6b6f0b92" pod="opendatahub/test-curl-7896-a7914d" Oct 29 22:59:51.006875 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:51.006791 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1348501a-589d-4577-9b79-4e4e6b6f0b92" path="/var/lib/kubelet/pods/1348501a-589d-4577-9b79-4e4e6b6f0b92/volumes" Oct 29 22:59:56.831643 ip-10-0-143-172 kubenswrapper[5736]: E1029 22:59:56.831575 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 22:59:56.970178 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:56.970137 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7"] Oct 29 22:59:56.970855 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:56.970835 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5848ab58-08cc-4029-8202-c81f5c9fb186" containerName="revoke-keys" Oct 29 22:59:56.970855 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:56.970856 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5848ab58-08cc-4029-8202-c81f5c9fb186" containerName="revoke-keys" Oct 29 22:59:56.971050 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:56.971036 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5848ab58-08cc-4029-8202-c81f5c9fb186" containerName="revoke-keys" Oct 29 22:59:56.974878 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:56.974858 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 22:59:56.995526 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:56.995498 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7"] Oct 29 22:59:57.000401 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.000375 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-df82ee\"" Oct 29 22:59:57.128652 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.117520 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84bpc\" (UniqueName: \"kubernetes.io/projected/954a1307-971f-41df-8c6b-1dc6c69b9495-kube-api-access-84bpc\") pod \"payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7\" (UID: \"954a1307-971f-41df-8c6b-1dc6c69b9495\") " pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 22:59:57.128652 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.117637 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954a1307-971f-41df-8c6b-1dc6c69b9495-plugins-config-volume\") pod \"payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7\" (UID: \"954a1307-971f-41df-8c6b-1dc6c69b9495\") " pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 22:59:57.230041 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.221863 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bpc\" (UniqueName: \"kubernetes.io/projected/954a1307-971f-41df-8c6b-1dc6c69b9495-kube-api-access-84bpc\") pod \"payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7\" (UID: \"954a1307-971f-41df-8c6b-1dc6c69b9495\") " pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 22:59:57.230041 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.221955 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954a1307-971f-41df-8c6b-1dc6c69b9495-plugins-config-volume\") pod \"payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7\" (UID: \"954a1307-971f-41df-8c6b-1dc6c69b9495\") " pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 22:59:57.230041 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.223116 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954a1307-971f-41df-8c6b-1dc6c69b9495-plugins-config-volume\") pod \"payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7\" (UID: \"954a1307-971f-41df-8c6b-1dc6c69b9495\") " pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 22:59:57.247315 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.247139 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bpc\" (UniqueName: \"kubernetes.io/projected/954a1307-971f-41df-8c6b-1dc6c69b9495-kube-api-access-84bpc\") pod \"payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7\" (UID: \"954a1307-971f-41df-8c6b-1dc6c69b9495\") " pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 22:59:57.288989 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.288953 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 22:59:57.618085 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.618047 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7"] Oct 29 22:59:57.619003 ip-10-0-143-172 kubenswrapper[5736]: W1029 22:59:57.618979 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954a1307_971f_41df_8c6b_1dc6c69b9495.slice/crio-2f65a9d777c910393016c662c3e14e8d8d3020a56f3f1d11177a161512dd6983 WatchSource:0}: Error finding container 2f65a9d777c910393016c662c3e14e8d8d3020a56f3f1d11177a161512dd6983: Status 404 returned error can't find the container with id 2f65a9d777c910393016c662c3e14e8d8d3020a56f3f1d11177a161512dd6983 Oct 29 22:59:57.790068 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.790024 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" event={"ID":"954a1307-971f-41df-8c6b-1dc6c69b9495","Type":"ContainerStarted","Data":"6fa252babc34f71dbd08df6dc2d4dcfdb38c2a1d6203e3d67eb3ad5ff621be24"} Oct 29 22:59:57.790287 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.790104 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" event={"ID":"954a1307-971f-41df-8c6b-1dc6c69b9495","Type":"ContainerStarted","Data":"2f65a9d777c910393016c662c3e14e8d8d3020a56f3f1d11177a161512dd6983"} Oct 29 22:59:57.790287 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.790232 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 22:59:57.826682 ip-10-0-143-172 kubenswrapper[5736]: I1029 22:59:57.826636 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" podStartSLOduration=1.8266245250000002 podStartE2EDuration="1.826624525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:57.825900362 +0000 UTC m=+994.941819967" watchObservedRunningTime="2026-10-29 22:59:57.826624525 +0000 UTC m=+994.942544127" Oct 29 23:00:00.156242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.156209 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7"] Oct 29 23:00:00.161301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.161279 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:00.170209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.170184 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl"] Oct 29 23:00:00.174314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.174296 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:00.174434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.174349 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k"] Oct 29 23:00:00.179113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.179093 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:00.180029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.180008 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b"] Oct 29 23:00:00.188254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.188222 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7\" (UID: \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:00.188374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.188301 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmd7\" (UniqueName: \"kubernetes.io/projected/7da91507-e921-458e-8985-83b0d34face6-kube-api-access-4cmd7\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k\" (UID: \"7da91507-e921-458e-8985-83b0d34face6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:00.188450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.188374 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da91507-e921-458e-8985-83b0d34face6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k\" (UID: \"7da91507-e921-458e-8985-83b0d34face6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:00.188450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.188426 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdcw\" (UniqueName: \"kubernetes.io/projected/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-kube-api-access-hmdcw\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7\" (UID: \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:00.188567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.188458 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4d6ece-0247-4643-8527-5c94a6085721-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl\" (UID: \"1e4d6ece-0247-4643-8527-5c94a6085721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:00.188687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.188671 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47g8\" (UniqueName: \"kubernetes.io/projected/1e4d6ece-0247-4643-8527-5c94a6085721-kube-api-access-n47g8\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl\" (UID: \"1e4d6ece-0247-4643-8527-5c94a6085721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:00.188866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.188849 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:00.189406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.189385 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:00:00.193294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.193271 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k"] Oct 29 23:00:00.198651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.198305 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:00.203294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.203243 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7"] Oct 29 23:00:00.209774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.209744 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx"] Oct 29 23:00:00.228773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.226902 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:00.228916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.228663 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k"] Oct 29 23:00:00.244708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.244659 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl"] Oct 29 23:00:00.253639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.252213 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd"] Oct 29 23:00:00.274843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.274814 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b"] Oct 29 23:00:00.274980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.274875 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k"] Oct 29 23:00:00.274980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.274905 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv"] Oct 29 23:00:00.276414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.276394 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:00.280831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.280808 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx"] Oct 29 23:00:00.281013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.280990 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd"] Oct 29 23:00:00.281132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.281088 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:00.286105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.286082 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv"] Oct 29 23:00:00.290442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.290413 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdcw\" (UniqueName: \"kubernetes.io/projected/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-kube-api-access-hmdcw\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7\" (UID: \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:00.290545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.290464 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcct7\" (UniqueName: \"kubernetes.io/projected/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-kube-api-access-tcct7\") pod \"maas-api-key-cleanup-29888580-fnq9b\" (UID: \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:00.290545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.290497 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4d6ece-0247-4643-8527-5c94a6085721-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl\" (UID: \"1e4d6ece-0247-4643-8527-5c94a6085721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:00.290545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.290537 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjw96\" (UniqueName: \"kubernetes.io/projected/b57724d2-cadd-4ae7-923e-ccb807500264-kube-api-access-jjw96\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv\" (UID: \"b57724d2-cadd-4ae7-923e-ccb807500264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:00.290879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.290851 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57724d2-cadd-4ae7-923e-ccb807500264-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv\" (UID: \"b57724d2-cadd-4ae7-923e-ccb807500264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:00.290982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.290944 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbc8\" (UniqueName: \"kubernetes.io/projected/965a9f37-2d56-48c1-a771-06abfa93ce06-kube-api-access-qnbc8\") pod \"maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx\" (UID: \"965a9f37-2d56-48c1-a771-06abfa93ce06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:00.291048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.290991 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvrbv\" (UniqueName: \"kubernetes.io/projected/abbd2d03-4855-4909-b72c-0b26ef149fda-kube-api-access-qvrbv\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd\" (UID: \"abbd2d03-4855-4909-b72c-0b26ef149fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:00.291112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291044 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47g8\" (UniqueName: \"kubernetes.io/projected/1e4d6ece-0247-4643-8527-5c94a6085721-kube-api-access-n47g8\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl\" (UID: \"1e4d6ece-0247-4643-8527-5c94a6085721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:00.291186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291160 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e55f73-e312-4276-94d0-97ded0d3184d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k\" (UID: \"32e55f73-e312-4276-94d0-97ded0d3184d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:00.291293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291196 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbd2d03-4855-4909-b72c-0b26ef149fda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd\" (UID: \"abbd2d03-4855-4909-b72c-0b26ef149fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:00.291293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291230 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a9f37-2d56-48c1-a771-06abfa93ce06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx\" (UID: \"965a9f37-2d56-48c1-a771-06abfa93ce06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:00.291293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291264 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7\" (UID: \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:00.291466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291302 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4d6ece-0247-4643-8527-5c94a6085721-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl\" (UID: \"1e4d6ece-0247-4643-8527-5c94a6085721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:00.291466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291305 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwvv\" (UniqueName: \"kubernetes.io/projected/32e55f73-e312-4276-94d0-97ded0d3184d-kube-api-access-pfwvv\") pod \"maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k\" (UID: \"32e55f73-e312-4276-94d0-97ded0d3184d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:00.291637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291478 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmd7\" (UniqueName: \"kubernetes.io/projected/7da91507-e921-458e-8985-83b0d34face6-kube-api-access-4cmd7\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k\" (UID: \"7da91507-e921-458e-8985-83b0d34face6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:00.291637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291563 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da91507-e921-458e-8985-83b0d34face6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k\" (UID: \"7da91507-e921-458e-8985-83b0d34face6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:00.291758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.291657 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-fnq9b\" (UID: \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:00.292239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.292214 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7\" (UID: \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:00.292658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.292633 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da91507-e921-458e-8985-83b0d34face6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k\" (UID: \"7da91507-e921-458e-8985-83b0d34face6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:00.311864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.311842 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdcw\" (UniqueName: \"kubernetes.io/projected/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-kube-api-access-hmdcw\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7\" (UID: \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:00.312078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.312057 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47g8\" (UniqueName: \"kubernetes.io/projected/1e4d6ece-0247-4643-8527-5c94a6085721-kube-api-access-n47g8\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl\" (UID: \"1e4d6ece-0247-4643-8527-5c94a6085721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:00.314846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.314825 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmd7\" (UniqueName: \"kubernetes.io/projected/7da91507-e921-458e-8985-83b0d34face6-kube-api-access-4cmd7\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k\" (UID: \"7da91507-e921-458e-8985-83b0d34face6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:00.393050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393020 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbc8\" (UniqueName: \"kubernetes.io/projected/965a9f37-2d56-48c1-a771-06abfa93ce06-kube-api-access-qnbc8\") pod \"maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx\" (UID: \"965a9f37-2d56-48c1-a771-06abfa93ce06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:00.393183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393060 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrbv\" (UniqueName: \"kubernetes.io/projected/abbd2d03-4855-4909-b72c-0b26ef149fda-kube-api-access-qvrbv\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd\" (UID: \"abbd2d03-4855-4909-b72c-0b26ef149fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:00.393183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393145 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e55f73-e312-4276-94d0-97ded0d3184d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k\" (UID: \"32e55f73-e312-4276-94d0-97ded0d3184d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:00.393183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393161 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbd2d03-4855-4909-b72c-0b26ef149fda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd\" (UID: \"abbd2d03-4855-4909-b72c-0b26ef149fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:00.393288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393192 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a9f37-2d56-48c1-a771-06abfa93ce06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx\" (UID: \"965a9f37-2d56-48c1-a771-06abfa93ce06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:00.393288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393237 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwvv\" (UniqueName: \"kubernetes.io/projected/32e55f73-e312-4276-94d0-97ded0d3184d-kube-api-access-pfwvv\") pod \"maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k\" (UID: \"32e55f73-e312-4276-94d0-97ded0d3184d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:00.393366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393349 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-fnq9b\" (UID: \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:00.393419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393405 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcct7\" (UniqueName: \"kubernetes.io/projected/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-kube-api-access-tcct7\") pod \"maas-api-key-cleanup-29888580-fnq9b\" (UID: \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:00.393471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393455 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjw96\" (UniqueName: \"kubernetes.io/projected/b57724d2-cadd-4ae7-923e-ccb807500264-kube-api-access-jjw96\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv\" (UID: \"b57724d2-cadd-4ae7-923e-ccb807500264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:00.393557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393536 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57724d2-cadd-4ae7-923e-ccb807500264-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv\" (UID: \"b57724d2-cadd-4ae7-923e-ccb807500264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:00.393943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393918 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbd2d03-4855-4909-b72c-0b26ef149fda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd\" (UID: \"abbd2d03-4855-4909-b72c-0b26ef149fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:00.394003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393979 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a9f37-2d56-48c1-a771-06abfa93ce06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx\" (UID: \"965a9f37-2d56-48c1-a771-06abfa93ce06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:00.394039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.393981 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e55f73-e312-4276-94d0-97ded0d3184d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k\" (UID: \"32e55f73-e312-4276-94d0-97ded0d3184d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:00.394080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.394065 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-fnq9b\" (UID: \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:00.394271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.394254 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57724d2-cadd-4ae7-923e-ccb807500264-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv\" (UID: \"b57724d2-cadd-4ae7-923e-ccb807500264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:00.414669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.414616 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwvv\" (UniqueName: \"kubernetes.io/projected/32e55f73-e312-4276-94d0-97ded0d3184d-kube-api-access-pfwvv\") pod \"maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k\" (UID: \"32e55f73-e312-4276-94d0-97ded0d3184d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:00.414669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.414654 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrbv\" (UniqueName: \"kubernetes.io/projected/abbd2d03-4855-4909-b72c-0b26ef149fda-kube-api-access-qvrbv\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd\" (UID: \"abbd2d03-4855-4909-b72c-0b26ef149fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:00.414795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.414657 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbc8\" (UniqueName: \"kubernetes.io/projected/965a9f37-2d56-48c1-a771-06abfa93ce06-kube-api-access-qnbc8\") pod \"maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx\" (UID: \"965a9f37-2d56-48c1-a771-06abfa93ce06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:00.414896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.414879 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjw96\" (UniqueName: \"kubernetes.io/projected/b57724d2-cadd-4ae7-923e-ccb807500264-kube-api-access-jjw96\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv\" (UID: \"b57724d2-cadd-4ae7-923e-ccb807500264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:00.415010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.414988 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcct7\" (UniqueName: \"kubernetes.io/projected/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-kube-api-access-tcct7\") pod \"maas-api-key-cleanup-29888580-fnq9b\" (UID: \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:00.477507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.477476 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:00.498138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.498110 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:00.525732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.525706 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:00.568536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.551947 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:00.574111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.571903 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:00.577185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.575935 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:00.597535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.596455 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:00.604518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.604498 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:00.793841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.793814 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7"] Oct 29 23:00:00.795155 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:00.795110 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32961c39_a2fe_4a02_a9e0_b6eeaebe1b88.slice/crio-a577654d81bbd60cce4eef8c9a5ab3d1cdb5b97005c2db0075870fd9402ee4d5 WatchSource:0}: Error finding container a577654d81bbd60cce4eef8c9a5ab3d1cdb5b97005c2db0075870fd9402ee4d5: Status 404 returned error can't find the container with id a577654d81bbd60cce4eef8c9a5ab3d1cdb5b97005c2db0075870fd9402ee4d5 Oct 29 23:00:00.826735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.826702 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" event={"ID":"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88","Type":"ContainerStarted","Data":"a577654d81bbd60cce4eef8c9a5ab3d1cdb5b97005c2db0075870fd9402ee4d5"} Oct 29 23:00:00.854004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.853979 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k"] Oct 29 23:00:00.855283 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:00.855259 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da91507_e921_458e_8985_83b0d34face6.slice/crio-99c71a4b911f1402e8fa732b92b58c9365d06c0e0027465ad88162276fb5be4c WatchSource:0}: Error finding container 99c71a4b911f1402e8fa732b92b58c9365d06c0e0027465ad88162276fb5be4c: Status 404 returned error can't find the container with id 99c71a4b911f1402e8fa732b92b58c9365d06c0e0027465ad88162276fb5be4c Oct 29 23:00:00.929122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.928477 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b"] Oct 29 23:00:00.931816 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:00.931781 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a8dffa_76b8_4abe_8dd4_102c367f5ea5.slice/crio-79453d1e17972c12e54af0b8b414cd0ee1195ccc07c272b3ce8b2396fab20723 WatchSource:0}: Error finding container 79453d1e17972c12e54af0b8b414cd0ee1195ccc07c272b3ce8b2396fab20723: Status 404 returned error can't find the container with id 79453d1e17972c12e54af0b8b414cd0ee1195ccc07c272b3ce8b2396fab20723 Oct 29 23:00:00.934388 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:00.934361 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e55f73_e312_4276_94d0_97ded0d3184d.slice/crio-03e6c81d2c7ec256c65c5388bb28559aeb47d5f40d1206d49008f08a1ffa6325 WatchSource:0}: Error finding container 03e6c81d2c7ec256c65c5388bb28559aeb47d5f40d1206d49008f08a1ffa6325: Status 404 returned error can't find the container with id 03e6c81d2c7ec256c65c5388bb28559aeb47d5f40d1206d49008f08a1ffa6325 Oct 29 23:00:00.937407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.937383 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k"] Oct 29 23:00:00.977923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:00.977895 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx"] Oct 29 23:00:01.007501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.007479 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd"] Oct 29 23:00:01.008430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.008409 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv"] Oct 29 23:00:01.014464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.014442 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl"] Oct 29 23:00:01.035009 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:01.034966 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965a9f37_2d56_48c1_a771_06abfa93ce06.slice/crio-f9000a0bd8d59ea4bc9cc702757cf2116156a7e714cbf09f62d4af304f2223fa WatchSource:0}: Error finding container f9000a0bd8d59ea4bc9cc702757cf2116156a7e714cbf09f62d4af304f2223fa: Status 404 returned error can't find the container with id f9000a0bd8d59ea4bc9cc702757cf2116156a7e714cbf09f62d4af304f2223fa Oct 29 23:00:01.743702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.743560 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 23:00:01.840942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.840844 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" event={"ID":"b57724d2-cadd-4ae7-923e-ccb807500264","Type":"ContainerStarted","Data":"e277f6ddb242ddae03d7950cb022f07455a7f61d48a31d2d28c067c3636245b0"} Oct 29 23:00:01.840942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.840926 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" event={"ID":"b57724d2-cadd-4ae7-923e-ccb807500264","Type":"ContainerStarted","Data":"cb7e4e2c9f927a485be7138b9ffe9827ea6551b175c46ab47c4572fb5eb043b9"} Oct 29 23:00:01.842881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.842846 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" event={"ID":"abbd2d03-4855-4909-b72c-0b26ef149fda","Type":"ContainerStarted","Data":"409c3dfbdeda1c39f27c309c175d85d41ec405038f7da9f1535fe40b77ffdd2c"} Oct 29 23:00:01.843019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.842907 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" event={"ID":"abbd2d03-4855-4909-b72c-0b26ef149fda","Type":"ContainerStarted","Data":"dfb587d1eb586bf1cbac4d77bf3f43c69a86b21e0469a1f05d7726f1382d8c5b"} Oct 29 23:00:01.844632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.844609 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" event={"ID":"32e55f73-e312-4276-94d0-97ded0d3184d","Type":"ContainerStarted","Data":"ad4b4fafcd49c0347532f91d66919bf092b7a4c36529025e53f4a5f717b5c697"} Oct 29 23:00:01.844827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.844801 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" event={"ID":"32e55f73-e312-4276-94d0-97ded0d3184d","Type":"ContainerStarted","Data":"03e6c81d2c7ec256c65c5388bb28559aeb47d5f40d1206d49008f08a1ffa6325"} Oct 29 23:00:01.846560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.846503 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" event={"ID":"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88","Type":"ContainerStarted","Data":"1772ff6e2bbadcd10852d6d4ccd524ee3aee9a6501278407687343b720428916"} Oct 29 23:00:01.849094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.849073 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" event={"ID":"1e4d6ece-0247-4643-8527-5c94a6085721","Type":"ContainerStarted","Data":"6e835c3676525e08166e184cda5c6854d69c9b41bf86b4c29f8d9ea696c13347"} Oct 29 23:00:01.849202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.849122 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" event={"ID":"1e4d6ece-0247-4643-8527-5c94a6085721","Type":"ContainerStarted","Data":"cab409c81d6e4d98bd727d3691669bcd1eeba571aad07a4d8803ece4e542efcf"} Oct 29 23:00:01.850798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.850768 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" event={"ID":"7da91507-e921-458e-8985-83b0d34face6","Type":"ContainerStarted","Data":"ca4d54866eb333635a5b3baa2327327c3fd3804e03469c8b12e126d87bf1a7e4"} Oct 29 23:00:01.850908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.850823 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" event={"ID":"7da91507-e921-458e-8985-83b0d34face6","Type":"ContainerStarted","Data":"99c71a4b911f1402e8fa732b92b58c9365d06c0e0027465ad88162276fb5be4c"} Oct 29 23:00:01.852623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.852564 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" event={"ID":"00a8dffa-76b8-4abe-8dd4-102c367f5ea5","Type":"ContainerStarted","Data":"554b6a954055d775345287e521fabec6e83b9dea5b2607e98ef4985cbdef9e29"} Oct 29 23:00:01.852713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.852643 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" event={"ID":"00a8dffa-76b8-4abe-8dd4-102c367f5ea5","Type":"ContainerStarted","Data":"79453d1e17972c12e54af0b8b414cd0ee1195ccc07c272b3ce8b2396fab20723"} Oct 29 23:00:01.854348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.854326 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" event={"ID":"965a9f37-2d56-48c1-a771-06abfa93ce06","Type":"ContainerStarted","Data":"e29332941f2121cd602ce7cc651d74601f3362635dec012aea6156247242db4b"} Oct 29 23:00:01.854446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.854368 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" event={"ID":"965a9f37-2d56-48c1-a771-06abfa93ce06","Type":"ContainerStarted","Data":"f9000a0bd8d59ea4bc9cc702757cf2116156a7e714cbf09f62d4af304f2223fa"} Oct 29 23:00:01.884069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.884014 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" podStartSLOduration=1.883996899 podStartE2EDuration="1.883996899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.878391829 +0000 UTC m=+998.994311434" watchObservedRunningTime="2026-10-29 23:00:01.883996899 +0000 UTC m=+998.999916498" Oct 29 23:00:01.920418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.920338 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" podStartSLOduration=1.9203211919999998 podStartE2EDuration="1.920321192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.919423388 +0000 UTC m=+999.035342995" watchObservedRunningTime="2026-10-29 23:00:01.920321192 +0000 UTC m=+999.036240791" Oct 29 23:00:01.960902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.960854 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" podStartSLOduration=1.960839282 podStartE2EDuration="1.960839282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.954725041 +0000 UTC m=+999.070644672" watchObservedRunningTime="2026-10-29 23:00:01.960839282 +0000 UTC m=+999.076758888" Oct 29 23:00:01.995951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:01.995892 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" podStartSLOduration=1.99587759 podStartE2EDuration="1.99587759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.992905384 +0000 UTC m=+999.108824990" watchObservedRunningTime="2026-10-29 23:00:01.99587759 +0000 UTC m=+999.111797195" Oct 29 23:00:02.032775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.032711 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" podStartSLOduration=2.032690778 podStartE2EDuration="2.032690778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.027788205 +0000 UTC m=+999.143707810" watchObservedRunningTime="2026-10-29 23:00:02.032690778 +0000 UTC m=+999.148610386" Oct 29 23:00:02.067832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.067623 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" podStartSLOduration=2.067563447 podStartE2EDuration="2.067563447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.066427272 +0000 UTC m=+999.182346875" watchObservedRunningTime="2026-10-29 23:00:02.067563447 +0000 UTC m=+999.183483052" Oct 29 23:00:02.105431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.105328 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" podStartSLOduration=2.105313308 podStartE2EDuration="2.105313308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.101630095 +0000 UTC m=+999.217549702" watchObservedRunningTime="2026-10-29 23:00:02.105313308 +0000 UTC m=+999.221232960" Oct 29 23:00:02.137102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.137054 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" podStartSLOduration=2.137038142 podStartE2EDuration="2.137038142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.136175502 +0000 UTC m=+999.252095108" watchObservedRunningTime="2026-10-29 23:00:02.137038142 +0000 UTC m=+999.252957749" Oct 29 23:00:02.863900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.863867 5736 generic.go:356] "Generic (PLEG): container finished" podID="b57724d2-cadd-4ae7-923e-ccb807500264" containerID="e277f6ddb242ddae03d7950cb022f07455a7f61d48a31d2d28c067c3636245b0" exitCode=60 Oct 29 23:00:02.864327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.863955 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" event={"ID":"b57724d2-cadd-4ae7-923e-ccb807500264","Type":"ContainerDied","Data":"e277f6ddb242ddae03d7950cb022f07455a7f61d48a31d2d28c067c3636245b0"} Oct 29 23:00:02.864327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.864272 5736 scope.go:122] "RemoveContainer" containerID="e277f6ddb242ddae03d7950cb022f07455a7f61d48a31d2d28c067c3636245b0" Oct 29 23:00:02.865703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.865682 5736 generic.go:356] "Generic (PLEG): container finished" podID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerID="409c3dfbdeda1c39f27c309c175d85d41ec405038f7da9f1535fe40b77ffdd2c" exitCode=60 Oct 29 23:00:02.865813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.865765 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" event={"ID":"abbd2d03-4855-4909-b72c-0b26ef149fda","Type":"ContainerDied","Data":"409c3dfbdeda1c39f27c309c175d85d41ec405038f7da9f1535fe40b77ffdd2c"} Oct 29 23:00:02.866008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.865992 5736 scope.go:122] "RemoveContainer" containerID="409c3dfbdeda1c39f27c309c175d85d41ec405038f7da9f1535fe40b77ffdd2c" Oct 29 23:00:02.867628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.867604 5736 generic.go:356] "Generic (PLEG): container finished" podID="32e55f73-e312-4276-94d0-97ded0d3184d" containerID="ad4b4fafcd49c0347532f91d66919bf092b7a4c36529025e53f4a5f717b5c697" exitCode=60 Oct 29 23:00:02.867738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.867637 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" event={"ID":"32e55f73-e312-4276-94d0-97ded0d3184d","Type":"ContainerDied","Data":"ad4b4fafcd49c0347532f91d66919bf092b7a4c36529025e53f4a5f717b5c697"} Oct 29 23:00:02.867940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.867865 5736 scope.go:122] "RemoveContainer" containerID="ad4b4fafcd49c0347532f91d66919bf092b7a4c36529025e53f4a5f717b5c697" Oct 29 23:00:02.869534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.869516 5736 generic.go:356] "Generic (PLEG): container finished" podID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerID="1772ff6e2bbadcd10852d6d4ccd524ee3aee9a6501278407687343b720428916" exitCode=60 Oct 29 23:00:02.869625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.869612 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" event={"ID":"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88","Type":"ContainerDied","Data":"1772ff6e2bbadcd10852d6d4ccd524ee3aee9a6501278407687343b720428916"} Oct 29 23:00:02.869818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.869803 5736 scope.go:122] "RemoveContainer" containerID="1772ff6e2bbadcd10852d6d4ccd524ee3aee9a6501278407687343b720428916" Oct 29 23:00:02.871192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.871173 5736 generic.go:356] "Generic (PLEG): container finished" podID="1e4d6ece-0247-4643-8527-5c94a6085721" containerID="6e835c3676525e08166e184cda5c6854d69c9b41bf86b4c29f8d9ea696c13347" exitCode=60 Oct 29 23:00:02.871284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.871247 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" event={"ID":"1e4d6ece-0247-4643-8527-5c94a6085721","Type":"ContainerDied","Data":"6e835c3676525e08166e184cda5c6854d69c9b41bf86b4c29f8d9ea696c13347"} Oct 29 23:00:02.871423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.871408 5736 scope.go:122] "RemoveContainer" containerID="6e835c3676525e08166e184cda5c6854d69c9b41bf86b4c29f8d9ea696c13347" Oct 29 23:00:02.872873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.872850 5736 generic.go:356] "Generic (PLEG): container finished" podID="7da91507-e921-458e-8985-83b0d34face6" containerID="ca4d54866eb333635a5b3baa2327327c3fd3804e03469c8b12e126d87bf1a7e4" exitCode=60 Oct 29 23:00:02.872948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.872873 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" event={"ID":"7da91507-e921-458e-8985-83b0d34face6","Type":"ContainerDied","Data":"ca4d54866eb333635a5b3baa2327327c3fd3804e03469c8b12e126d87bf1a7e4"} Oct 29 23:00:02.873126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.873102 5736 scope.go:122] "RemoveContainer" containerID="ca4d54866eb333635a5b3baa2327327c3fd3804e03469c8b12e126d87bf1a7e4" Oct 29 23:00:02.874708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.874683 5736 generic.go:356] "Generic (PLEG): container finished" podID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerID="554b6a954055d775345287e521fabec6e83b9dea5b2607e98ef4985cbdef9e29" exitCode=60 Oct 29 23:00:02.874789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.874759 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" event={"ID":"00a8dffa-76b8-4abe-8dd4-102c367f5ea5","Type":"ContainerDied","Data":"554b6a954055d775345287e521fabec6e83b9dea5b2607e98ef4985cbdef9e29"} Oct 29 23:00:02.874938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.874919 5736 scope.go:122] "RemoveContainer" containerID="554b6a954055d775345287e521fabec6e83b9dea5b2607e98ef4985cbdef9e29" Oct 29 23:00:02.876570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.876547 5736 generic.go:356] "Generic (PLEG): container finished" podID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerID="e29332941f2121cd602ce7cc651d74601f3362635dec012aea6156247242db4b" exitCode=60 Oct 29 23:00:02.876685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.876621 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" event={"ID":"965a9f37-2d56-48c1-a771-06abfa93ce06","Type":"ContainerDied","Data":"e29332941f2121cd602ce7cc651d74601f3362635dec012aea6156247242db4b"} Oct 29 23:00:02.876955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:02.876933 5736 scope.go:122] "RemoveContainer" containerID="e29332941f2121cd602ce7cc651d74601f3362635dec012aea6156247242db4b" Oct 29 23:00:03.887046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:03.886964 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" event={"ID":"abbd2d03-4855-4909-b72c-0b26ef149fda","Type":"ContainerStarted","Data":"75dff49093ca6e8493e09948dd77908964bb74f175615dbf9c349a252c0a83fe"} Oct 29 23:00:03.888854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:03.888819 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" event={"ID":"32e55f73-e312-4276-94d0-97ded0d3184d","Type":"ContainerStarted","Data":"a14a34cb72253a98c75c84048873df7d1b1c8fa3108827116485e1544d31e3c9"} Oct 29 23:00:03.890634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:03.890607 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" event={"ID":"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88","Type":"ContainerStarted","Data":"24e7548688db82941c93a74a64813b3016d410a2c1e4357ec42c6781f02ea5d8"} Oct 29 23:00:03.892546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:03.892524 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" event={"ID":"1e4d6ece-0247-4643-8527-5c94a6085721","Type":"ContainerStarted","Data":"fa8bf74442efbea69d049daf2d6216d75c6b0287a4a281d96ae9196d68e43329"} Oct 29 23:00:03.894574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:03.894547 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" event={"ID":"7da91507-e921-458e-8985-83b0d34face6","Type":"ContainerStarted","Data":"91fa8fdb574fb9d4d160c22bd0bb3a1ae0ba938e4187d972872eb40733280266"} Oct 29 23:00:03.896634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:03.896607 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" event={"ID":"00a8dffa-76b8-4abe-8dd4-102c367f5ea5","Type":"ContainerStarted","Data":"04c3c0023a67521386e90ee0830bb1b2026c691e57a8213806b8c9e4465ef797"} Oct 29 23:00:03.898618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:03.898564 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" event={"ID":"965a9f37-2d56-48c1-a771-06abfa93ce06","Type":"ContainerStarted","Data":"9eab3a29bc4093dc754ac0f611ff3cda9e293fa84ef2e5531a734f3bf069315b"} Oct 29 23:00:03.901397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:03.901375 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" event={"ID":"b57724d2-cadd-4ae7-923e-ccb807500264","Type":"ContainerStarted","Data":"b27a92ba4f3dd8f3c48993f6cc2e8688892e207b8aee6664f58c3c7b4053f1c7"} Oct 29 23:00:04.922496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.922456 5736 generic.go:356] "Generic (PLEG): container finished" podID="32e55f73-e312-4276-94d0-97ded0d3184d" containerID="a14a34cb72253a98c75c84048873df7d1b1c8fa3108827116485e1544d31e3c9" exitCode=60 Oct 29 23:00:04.923025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.922639 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" event={"ID":"32e55f73-e312-4276-94d0-97ded0d3184d","Type":"ContainerDied","Data":"a14a34cb72253a98c75c84048873df7d1b1c8fa3108827116485e1544d31e3c9"} Oct 29 23:00:04.923025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.922756 5736 scope.go:122] "RemoveContainer" containerID="ad4b4fafcd49c0347532f91d66919bf092b7a4c36529025e53f4a5f717b5c697" Oct 29 23:00:04.923429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.923408 5736 scope.go:122] "RemoveContainer" containerID="a14a34cb72253a98c75c84048873df7d1b1c8fa3108827116485e1544d31e3c9" Oct 29 23:00:04.924357 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:04.924054 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k_odh-ai-gateway-infra(32e55f73-e312-4276-94d0-97ded0d3184d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" Oct 29 23:00:04.927989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.927960 5736 generic.go:356] "Generic (PLEG): container finished" podID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerID="24e7548688db82941c93a74a64813b3016d410a2c1e4357ec42c6781f02ea5d8" exitCode=60 Oct 29 23:00:04.928130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.928075 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" event={"ID":"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88","Type":"ContainerDied","Data":"24e7548688db82941c93a74a64813b3016d410a2c1e4357ec42c6781f02ea5d8"} Oct 29 23:00:04.928755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.928563 5736 scope.go:122] "RemoveContainer" containerID="24e7548688db82941c93a74a64813b3016d410a2c1e4357ec42c6781f02ea5d8" Oct 29 23:00:04.929137 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:04.929083 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7_odh-ai-gateway-infra(32961c39-a2fe-4a02-a9e0-b6eeaebe1b88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" Oct 29 23:00:04.931077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.931052 5736 generic.go:356] "Generic (PLEG): container finished" podID="1e4d6ece-0247-4643-8527-5c94a6085721" containerID="fa8bf74442efbea69d049daf2d6216d75c6b0287a4a281d96ae9196d68e43329" exitCode=60 Oct 29 23:00:04.931204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.931154 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" event={"ID":"1e4d6ece-0247-4643-8527-5c94a6085721","Type":"ContainerDied","Data":"fa8bf74442efbea69d049daf2d6216d75c6b0287a4a281d96ae9196d68e43329"} Oct 29 23:00:04.931677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.931655 5736 scope.go:122] "RemoveContainer" containerID="fa8bf74442efbea69d049daf2d6216d75c6b0287a4a281d96ae9196d68e43329" Oct 29 23:00:04.932214 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:04.932188 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl_odh-ai-gateway-infra(1e4d6ece-0247-4643-8527-5c94a6085721)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" Oct 29 23:00:04.935064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.935034 5736 generic.go:356] "Generic (PLEG): container finished" podID="7da91507-e921-458e-8985-83b0d34face6" containerID="91fa8fdb574fb9d4d160c22bd0bb3a1ae0ba938e4187d972872eb40733280266" exitCode=60 Oct 29 23:00:04.935270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.935244 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" event={"ID":"7da91507-e921-458e-8985-83b0d34face6","Type":"ContainerDied","Data":"91fa8fdb574fb9d4d160c22bd0bb3a1ae0ba938e4187d972872eb40733280266"} Oct 29 23:00:04.935524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.935500 5736 scope.go:122] "RemoveContainer" containerID="91fa8fdb574fb9d4d160c22bd0bb3a1ae0ba938e4187d972872eb40733280266" Oct 29 23:00:04.936229 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:04.936200 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k_odh-ai-gateway-infra(7da91507-e921-458e-8985-83b0d34face6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" podUID="7da91507-e921-458e-8985-83b0d34face6" Oct 29 23:00:04.938065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.938038 5736 generic.go:356] "Generic (PLEG): container finished" podID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerID="04c3c0023a67521386e90ee0830bb1b2026c691e57a8213806b8c9e4465ef797" exitCode=60 Oct 29 23:00:04.938260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.938241 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" event={"ID":"00a8dffa-76b8-4abe-8dd4-102c367f5ea5","Type":"ContainerDied","Data":"04c3c0023a67521386e90ee0830bb1b2026c691e57a8213806b8c9e4465ef797"} Oct 29 23:00:04.938763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.938741 5736 scope.go:122] "RemoveContainer" containerID="04c3c0023a67521386e90ee0830bb1b2026c691e57a8213806b8c9e4465ef797" Oct 29 23:00:04.940358 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:04.940326 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-fnq9b_odh-ai-gateway-infra(00a8dffa-76b8-4abe-8dd4-102c367f5ea5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" Oct 29 23:00:04.940634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.940571 5736 generic.go:356] "Generic (PLEG): container finished" podID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerID="9eab3a29bc4093dc754ac0f611ff3cda9e293fa84ef2e5531a734f3bf069315b" exitCode=60 Oct 29 23:00:04.942155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.940637 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" event={"ID":"965a9f37-2d56-48c1-a771-06abfa93ce06","Type":"ContainerDied","Data":"9eab3a29bc4093dc754ac0f611ff3cda9e293fa84ef2e5531a734f3bf069315b"} Oct 29 23:00:04.942155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.941069 5736 scope.go:122] "RemoveContainer" containerID="9eab3a29bc4093dc754ac0f611ff3cda9e293fa84ef2e5531a734f3bf069315b" Oct 29 23:00:04.942155 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:04.941425 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx_odh-ai-gateway-infra(965a9f37-2d56-48c1-a771-06abfa93ce06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" Oct 29 23:00:04.949818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.949790 5736 generic.go:356] "Generic (PLEG): container finished" podID="b57724d2-cadd-4ae7-923e-ccb807500264" containerID="b27a92ba4f3dd8f3c48993f6cc2e8688892e207b8aee6664f58c3c7b4053f1c7" exitCode=60 Oct 29 23:00:04.950032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.949878 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" event={"ID":"b57724d2-cadd-4ae7-923e-ccb807500264","Type":"ContainerDied","Data":"b27a92ba4f3dd8f3c48993f6cc2e8688892e207b8aee6664f58c3c7b4053f1c7"} Oct 29 23:00:04.950102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.950084 5736 scope.go:122] "RemoveContainer" containerID="b27a92ba4f3dd8f3c48993f6cc2e8688892e207b8aee6664f58c3c7b4053f1c7" Oct 29 23:00:04.950379 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:04.950353 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv_odh-ai-gateway-infra(b57724d2-cadd-4ae7-923e-ccb807500264)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" Oct 29 23:00:04.950844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.950826 5736 scope.go:122] "RemoveContainer" containerID="1772ff6e2bbadcd10852d6d4ccd524ee3aee9a6501278407687343b720428916" Oct 29 23:00:04.952260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.951965 5736 generic.go:356] "Generic (PLEG): container finished" podID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerID="75dff49093ca6e8493e09948dd77908964bb74f175615dbf9c349a252c0a83fe" exitCode=60 Oct 29 23:00:04.952260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.952020 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" event={"ID":"abbd2d03-4855-4909-b72c-0b26ef149fda","Type":"ContainerDied","Data":"75dff49093ca6e8493e09948dd77908964bb74f175615dbf9c349a252c0a83fe"} Oct 29 23:00:04.952260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.952256 5736 scope.go:122] "RemoveContainer" containerID="75dff49093ca6e8493e09948dd77908964bb74f175615dbf9c349a252c0a83fe" Oct 29 23:00:04.952482 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:04.952460 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd_odh-ai-gateway-infra(abbd2d03-4855-4909-b72c-0b26ef149fda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" Oct 29 23:00:04.969924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.969900 5736 scope.go:122] "RemoveContainer" containerID="6e835c3676525e08166e184cda5c6854d69c9b41bf86b4c29f8d9ea696c13347" Oct 29 23:00:04.987713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:04.987601 5736 scope.go:122] "RemoveContainer" containerID="ca4d54866eb333635a5b3baa2327327c3fd3804e03469c8b12e126d87bf1a7e4" Oct 29 23:00:05.006190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.006160 5736 scope.go:122] "RemoveContainer" containerID="554b6a954055d775345287e521fabec6e83b9dea5b2607e98ef4985cbdef9e29" Oct 29 23:00:05.021716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.021694 5736 scope.go:122] "RemoveContainer" containerID="e29332941f2121cd602ce7cc651d74601f3362635dec012aea6156247242db4b" Oct 29 23:00:05.037410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.037385 5736 scope.go:122] "RemoveContainer" containerID="e277f6ddb242ddae03d7950cb022f07455a7f61d48a31d2d28c067c3636245b0" Oct 29 23:00:05.053662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.053636 5736 scope.go:122] "RemoveContainer" containerID="409c3dfbdeda1c39f27c309c175d85d41ec405038f7da9f1535fe40b77ffdd2c" Oct 29 23:00:05.963079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.963042 5736 scope.go:122] "RemoveContainer" containerID="a14a34cb72253a98c75c84048873df7d1b1c8fa3108827116485e1544d31e3c9" Oct 29 23:00:05.963568 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:05.963460 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k_odh-ai-gateway-infra(32e55f73-e312-4276-94d0-97ded0d3184d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" Oct 29 23:00:05.964858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.964832 5736 scope.go:122] "RemoveContainer" containerID="24e7548688db82941c93a74a64813b3016d410a2c1e4357ec42c6781f02ea5d8" Oct 29 23:00:05.965209 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:05.965186 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7_odh-ai-gateway-infra(32961c39-a2fe-4a02-a9e0-b6eeaebe1b88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" Oct 29 23:00:05.966591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.966557 5736 scope.go:122] "RemoveContainer" containerID="fa8bf74442efbea69d049daf2d6216d75c6b0287a4a281d96ae9196d68e43329" Oct 29 23:00:05.966940 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:05.966916 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl_odh-ai-gateway-infra(1e4d6ece-0247-4643-8527-5c94a6085721)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" Oct 29 23:00:05.968217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.968199 5736 scope.go:122] "RemoveContainer" containerID="91fa8fdb574fb9d4d160c22bd0bb3a1ae0ba938e4187d972872eb40733280266" Oct 29 23:00:05.968489 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:05.968470 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k_odh-ai-gateway-infra(7da91507-e921-458e-8985-83b0d34face6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" podUID="7da91507-e921-458e-8985-83b0d34face6" Oct 29 23:00:05.969890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.969872 5736 scope.go:122] "RemoveContainer" containerID="04c3c0023a67521386e90ee0830bb1b2026c691e57a8213806b8c9e4465ef797" Oct 29 23:00:05.970154 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:05.970133 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-fnq9b_odh-ai-gateway-infra(00a8dffa-76b8-4abe-8dd4-102c367f5ea5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" Oct 29 23:00:05.971541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.971505 5736 scope.go:122] "RemoveContainer" containerID="9eab3a29bc4093dc754ac0f611ff3cda9e293fa84ef2e5531a734f3bf069315b" Oct 29 23:00:05.971827 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:05.971805 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx_odh-ai-gateway-infra(965a9f37-2d56-48c1-a771-06abfa93ce06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" Oct 29 23:00:05.973141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.973122 5736 scope.go:122] "RemoveContainer" containerID="b27a92ba4f3dd8f3c48993f6cc2e8688892e207b8aee6664f58c3c7b4053f1c7" Oct 29 23:00:05.973358 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:05.973337 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv_odh-ai-gateway-infra(b57724d2-cadd-4ae7-923e-ccb807500264)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" Oct 29 23:00:05.974826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:05.974809 5736 scope.go:122] "RemoveContainer" containerID="75dff49093ca6e8493e09948dd77908964bb74f175615dbf9c349a252c0a83fe" Oct 29 23:00:05.975018 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:05.975001 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd_odh-ai-gateway-infra(abbd2d03-4855-4909-b72c-0b26ef149fda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" Oct 29 23:00:07.036779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:07.036740 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc"] Oct 29 23:00:07.071346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:07.071307 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d5d776-87jlc"] Oct 29 23:00:08.140760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.140424 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2"] Oct 29 23:00:08.146758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.146733 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.163053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.163028 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7fa011-openshift-default-dockercfg-p6vq7\"" Oct 29 23:00:08.186498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.186463 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2"] Oct 29 23:00:08.235267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.235233 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-envoy\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.235463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.235343 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-socket\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.235534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.235479 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-certs\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.235534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.235513 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-data\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.235638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.235565 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxzfc\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-kube-api-access-mxzfc\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.235698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.235678 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-token\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.235749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.235710 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istiod-ca-cert\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.235797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.235749 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-podinfo\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.235869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.235855 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-credential-socket\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.338601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.338148 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-socket\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.338601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.338275 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-certs\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.338601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.338304 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-data\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.338601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.338331 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxzfc\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-kube-api-access-mxzfc\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.338601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.338431 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-token\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.338601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.338466 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istiod-ca-cert\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.338601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.338523 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-podinfo\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.339109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.338755 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-credential-socket\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.339109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.339004 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-envoy\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.359608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.339913 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-certs\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.359608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.340301 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-socket\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.359608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.341018 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istiod-ca-cert\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.359608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.341284 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-credential-socket\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.359608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.341508 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-data\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.372602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.369484 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-podinfo\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.385629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.380700 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-token\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.385629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.385113 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-envoy\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.426656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.417353 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxzfc\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-kube-api-access-mxzfc\") pod \"e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.465601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.464986 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:08.803061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.802930 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 23:00:08.808989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.808962 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2"] Oct 29 23:00:08.812498 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:08.812465 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57fe8cc_c1f6_4f0d_8a0e_f56430402465.slice/crio-cbc1e67dc1984dd7961eacbd2811a62df005d6a78114fc2150bc5b8215714c73 WatchSource:0}: Error finding container cbc1e67dc1984dd7961eacbd2811a62df005d6a78114fc2150bc5b8215714c73: Status 404 returned error can't find the container with id cbc1e67dc1984dd7961eacbd2811a62df005d6a78114fc2150bc5b8215714c73 Oct 29 23:00:08.815799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.815764 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:08.815901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.815874 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:08.815974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:08.815920 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:09.017601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.017254 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c74229-1968-40a8-b6a8-017e59e03867" path="/var/lib/kubelet/pods/c9c74229-1968-40a8-b6a8-017e59e03867/volumes" Oct 29 23:00:09.021570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.021528 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" event={"ID":"c57fe8cc-c1f6-4f0d-8a0e-f56430402465","Type":"ContainerStarted","Data":"9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b"} Oct 29 23:00:09.021767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.021670 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" event={"ID":"c57fe8cc-c1f6-4f0d-8a0e-f56430402465","Type":"ContainerStarted","Data":"cbc1e67dc1984dd7961eacbd2811a62df005d6a78114fc2150bc5b8215714c73"} Oct 29 23:00:09.067112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.067007 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" podStartSLOduration=1.066992853 podStartE2EDuration="1.066992853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:09.06224382 +0000 UTC m=+1006.178163425" watchObservedRunningTime="2026-10-29 23:00:09.066992853 +0000 UTC m=+1006.182912459" Oct 29 23:00:09.230432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.230396 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g"] Oct 29 23:00:09.236200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.236180 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.252496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.252474 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-475876-serving-cert\"" Oct 29 23:00:09.261599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.259335 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g"] Oct 29 23:00:09.363488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.362676 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.363488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.362768 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94cn9\" (UniqueName: \"kubernetes.io/projected/d42eb596-4e68-4419-94d7-733d66c152c1-kube-api-access-94cn9\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.363488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.362932 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-metrics-certs\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.363488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.363100 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42eb596-4e68-4419-94d7-733d66c152c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.464842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.464806 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-metrics-certs\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.465042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.464993 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42eb596-4e68-4419-94d7-733d66c152c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.465132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.465098 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.465279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.465163 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94cn9\" (UniqueName: \"kubernetes.io/projected/d42eb596-4e68-4419-94d7-733d66c152c1-kube-api-access-94cn9\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.465937 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:09.465912 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-serving-cert: secret "maas-api-e2e-ipp-a-475876-serving-cert" not found Oct 29 23:00:09.466051 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:09.466035 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls podName:d42eb596-4e68-4419-94d7-733d66c152c1 nodeName:}" failed. No retries permitted until 2026-10-29 23:00:09.966011202 +0000 UTC m=+1007.081930809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls") pod "maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" (UID: "d42eb596-4e68-4419-94d7-733d66c152c1") : secret "maas-api-e2e-ipp-a-475876-serving-cert" not found Oct 29 23:00:09.466119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.466062 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:09.469621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.466766 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42eb596-4e68-4419-94d7-733d66c152c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.470735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.470710 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-metrics-certs\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.473089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.473069 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:09.491913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.491879 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94cn9\" (UniqueName: \"kubernetes.io/projected/d42eb596-4e68-4419-94d7-733d66c152c1-kube-api-access-94cn9\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.976241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.976204 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:09.978704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:09.978661 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:10.032868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:10.032832 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:10.034028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:10.034002 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:00:10.152840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:10.152810 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:10.608113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:10.608079 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g"] Oct 29 23:00:11.048058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:11.048021 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" event={"ID":"d42eb596-4e68-4419-94d7-733d66c152c1","Type":"ContainerStarted","Data":"0e34b385484752f0fceb46425b943f1152872f176e39b86c62644e45de43ce80"} Oct 29 23:00:12.044282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:12.044248 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb"] Oct 29 23:00:12.054486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:12.054459 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fffb5d-hjsmb"] Oct 29 23:00:12.060214 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:12.060180 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" event={"ID":"d42eb596-4e68-4419-94d7-733d66c152c1","Type":"ContainerStarted","Data":"3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c"} Oct 29 23:00:12.060358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:12.060266 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:12.104145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:12.104097 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" podStartSLOduration=2.7174913309999997 podStartE2EDuration="3.104083856s" totalImagesPullingTime="386.592525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:12.099344993 +0000 UTC m=+1009.215264600" watchObservedRunningTime="2026-10-29 23:00:12.104083856 +0000 UTC m=+1009.220003500" Oct 29 23:00:13.005960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.005925 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985a621f-26f7-40c1-a406-551845957ee0" path="/var/lib/kubelet/pods/985a621f-26f7-40c1-a406-551845957ee0/volumes" Oct 29 23:00:13.350650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.347952 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84"] Oct 29 23:00:13.354908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.354871 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:13.372184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.372150 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84"] Oct 29 23:00:13.374852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.374811 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7fa011\"" Oct 29 23:00:13.375008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.374989 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7fa011-dockercfg-ccqnn\"" Oct 29 23:00:13.417438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.417412 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp"] Oct 29 23:00:13.422450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.422430 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:13.439996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.439972 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp"] Oct 29 23:00:13.453249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.453223 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c3f3c5-b591-4414-9be0-450aff82223d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84\" (UID: \"c0c3f3c5-b591-4414-9be0-450aff82223d\") " pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:13.455828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.455795 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27742035-f9a1-4bf3-aacf-27483e0c4cad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp\" (UID: \"27742035-f9a1-4bf3-aacf-27483e0c4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:13.455942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.455880 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcxb\" (UniqueName: \"kubernetes.io/projected/c0c3f3c5-b591-4414-9be0-450aff82223d-kube-api-access-9fcxb\") pod \"payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84\" (UID: \"c0c3f3c5-b591-4414-9be0-450aff82223d\") " pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:13.459075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.459037 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cg9v\" (UniqueName: \"kubernetes.io/projected/27742035-f9a1-4bf3-aacf-27483e0c4cad-kube-api-access-6cg9v\") pod \"payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp\" (UID: \"27742035-f9a1-4bf3-aacf-27483e0c4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:13.646656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.620684 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cg9v\" (UniqueName: \"kubernetes.io/projected/27742035-f9a1-4bf3-aacf-27483e0c4cad-kube-api-access-6cg9v\") pod \"payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp\" (UID: \"27742035-f9a1-4bf3-aacf-27483e0c4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:13.676694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.665960 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c3f3c5-b591-4414-9be0-450aff82223d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84\" (UID: \"c0c3f3c5-b591-4414-9be0-450aff82223d\") " pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:13.676694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.666100 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27742035-f9a1-4bf3-aacf-27483e0c4cad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp\" (UID: \"27742035-f9a1-4bf3-aacf-27483e0c4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:13.676694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.666160 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcxb\" (UniqueName: \"kubernetes.io/projected/c0c3f3c5-b591-4414-9be0-450aff82223d-kube-api-access-9fcxb\") pod \"payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84\" (UID: \"c0c3f3c5-b591-4414-9be0-450aff82223d\") " pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:13.676694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.667950 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c3f3c5-b591-4414-9be0-450aff82223d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84\" (UID: \"c0c3f3c5-b591-4414-9be0-450aff82223d\") " pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:13.676694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.669096 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27742035-f9a1-4bf3-aacf-27483e0c4cad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp\" (UID: \"27742035-f9a1-4bf3-aacf-27483e0c4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:13.726201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.721757 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cg9v\" (UniqueName: \"kubernetes.io/projected/27742035-f9a1-4bf3-aacf-27483e0c4cad-kube-api-access-6cg9v\") pod \"payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp\" (UID: \"27742035-f9a1-4bf3-aacf-27483e0c4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:13.726201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.724736 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcxb\" (UniqueName: \"kubernetes.io/projected/c0c3f3c5-b591-4414-9be0-450aff82223d-kube-api-access-9fcxb\") pod \"payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84\" (UID: \"c0c3f3c5-b591-4414-9be0-450aff82223d\") " pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:13.743432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.743405 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:13.976138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:13.976104 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:14.069668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:14.069636 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp"] Oct 29 23:00:14.075378 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:14.075350 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27742035_f9a1_4bf3_aacf_27483e0c4cad.slice/crio-491897e82b69b89f2d0b3f8d486c2065d50fb3716df1f16e62371e407891bc8f WatchSource:0}: Error finding container 491897e82b69b89f2d0b3f8d486c2065d50fb3716df1f16e62371e407891bc8f: Status 404 returned error can't find the container with id 491897e82b69b89f2d0b3f8d486c2065d50fb3716df1f16e62371e407891bc8f Oct 29 23:00:14.293804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:14.293771 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84"] Oct 29 23:00:14.294486 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:14.294456 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c3f3c5_b591_4414_9be0_450aff82223d.slice/crio-2d577b188d1ee878ef4e8406dbe8c652b5bdc1223f5c2d4a2816a0bd14f80b2b WatchSource:0}: Error finding container 2d577b188d1ee878ef4e8406dbe8c652b5bdc1223f5c2d4a2816a0bd14f80b2b: Status 404 returned error can't find the container with id 2d577b188d1ee878ef4e8406dbe8c652b5bdc1223f5c2d4a2816a0bd14f80b2b Oct 29 23:00:14.996713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:14.996680 5736 scope.go:122] "RemoveContainer" containerID="9eab3a29bc4093dc754ac0f611ff3cda9e293fa84ef2e5531a734f3bf069315b" Oct 29 23:00:15.087952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.087912 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" event={"ID":"27742035-f9a1-4bf3-aacf-27483e0c4cad","Type":"ContainerStarted","Data":"824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635"} Oct 29 23:00:15.088056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.087977 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" event={"ID":"27742035-f9a1-4bf3-aacf-27483e0c4cad","Type":"ContainerStarted","Data":"491897e82b69b89f2d0b3f8d486c2065d50fb3716df1f16e62371e407891bc8f"} Oct 29 23:00:15.098820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.088725 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:15.098820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.095244 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" event={"ID":"c0c3f3c5-b591-4414-9be0-450aff82223d","Type":"ContainerStarted","Data":"8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb"} Oct 29 23:00:15.098820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.095330 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" event={"ID":"c0c3f3c5-b591-4414-9be0-450aff82223d","Type":"ContainerStarted","Data":"2d577b188d1ee878ef4e8406dbe8c652b5bdc1223f5c2d4a2816a0bd14f80b2b"} Oct 29 23:00:15.098820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.095637 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:15.127438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.127387 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" podStartSLOduration=2.127371881 podStartE2EDuration="2.127371881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:15.124931711 +0000 UTC m=+1012.240851318" watchObservedRunningTime="2026-10-29 23:00:15.127371881 +0000 UTC m=+1012.243291488" Oct 29 23:00:15.162675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.162621 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" podStartSLOduration=2.162608401 podStartE2EDuration="2.162608401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:15.16048715 +0000 UTC m=+1012.276406756" watchObservedRunningTime="2026-10-29 23:00:15.162608401 +0000 UTC m=+1012.278528007" Oct 29 23:00:15.997404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.997364 5736 scope.go:122] "RemoveContainer" containerID="b27a92ba4f3dd8f3c48993f6cc2e8688892e207b8aee6664f58c3c7b4053f1c7" Oct 29 23:00:15.997961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.997641 5736 scope.go:122] "RemoveContainer" containerID="91fa8fdb574fb9d4d160c22bd0bb3a1ae0ba938e4187d972872eb40733280266" Oct 29 23:00:15.997961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.997869 5736 scope.go:122] "RemoveContainer" containerID="24e7548688db82941c93a74a64813b3016d410a2c1e4357ec42c6781f02ea5d8" Oct 29 23:00:15.998092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:15.998070 5736 scope.go:122] "RemoveContainer" containerID="fa8bf74442efbea69d049daf2d6216d75c6b0287a4a281d96ae9196d68e43329" Oct 29 23:00:16.105505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:16.105472 5736 generic.go:356] "Generic (PLEG): container finished" podID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerID="dcb303295db6f31d1bb8843a788a82b2b71a1bfbea5a5fe08df157030496e294" exitCode=60 Oct 29 23:00:16.105610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:16.105510 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" event={"ID":"965a9f37-2d56-48c1-a771-06abfa93ce06","Type":"ContainerDied","Data":"dcb303295db6f31d1bb8843a788a82b2b71a1bfbea5a5fe08df157030496e294"} Oct 29 23:00:16.105610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:16.105575 5736 scope.go:122] "RemoveContainer" containerID="9eab3a29bc4093dc754ac0f611ff3cda9e293fa84ef2e5531a734f3bf069315b" Oct 29 23:00:16.106041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:16.106022 5736 scope.go:122] "RemoveContainer" containerID="dcb303295db6f31d1bb8843a788a82b2b71a1bfbea5a5fe08df157030496e294" Oct 29 23:00:16.106414 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:16.106390 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx_odh-ai-gateway-infra(965a9f37-2d56-48c1-a771-06abfa93ce06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" Oct 29 23:00:16.997275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:16.997249 5736 scope.go:122] "RemoveContainer" containerID="75dff49093ca6e8493e09948dd77908964bb74f175615dbf9c349a252c0a83fe" Oct 29 23:00:16.997818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:16.997795 5736 scope.go:122] "RemoveContainer" containerID="a14a34cb72253a98c75c84048873df7d1b1c8fa3108827116485e1544d31e3c9" Oct 29 23:00:17.115409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.115375 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" event={"ID":"b57724d2-cadd-4ae7-923e-ccb807500264","Type":"ContainerStarted","Data":"230010e160ff8d1fcc7c779844cb56faa0c2a7086e1be005f900ef0db123e242"} Oct 29 23:00:17.117366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.117339 5736 generic.go:356] "Generic (PLEG): container finished" podID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerID="de1158d4b5842c413fa9724ce614c3ec31bbbe4f1c3aacf3aec138ed817786a8" exitCode=60 Oct 29 23:00:17.117488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.117406 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" event={"ID":"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88","Type":"ContainerDied","Data":"de1158d4b5842c413fa9724ce614c3ec31bbbe4f1c3aacf3aec138ed817786a8"} Oct 29 23:00:17.117488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.117456 5736 scope.go:122] "RemoveContainer" containerID="24e7548688db82941c93a74a64813b3016d410a2c1e4357ec42c6781f02ea5d8" Oct 29 23:00:17.117860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.117844 5736 scope.go:122] "RemoveContainer" containerID="de1158d4b5842c413fa9724ce614c3ec31bbbe4f1c3aacf3aec138ed817786a8" Oct 29 23:00:17.118215 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:17.118193 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7_odh-ai-gateway-infra(32961c39-a2fe-4a02-a9e0-b6eeaebe1b88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" Oct 29 23:00:17.119333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.119309 5736 generic.go:356] "Generic (PLEG): container finished" podID="1e4d6ece-0247-4643-8527-5c94a6085721" containerID="3c8b1a3928a82de2b2c8c9ca64563915a475c59d604b8769dfe0d925c76f6303" exitCode=60 Oct 29 23:00:17.119405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.119375 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" event={"ID":"1e4d6ece-0247-4643-8527-5c94a6085721","Type":"ContainerDied","Data":"3c8b1a3928a82de2b2c8c9ca64563915a475c59d604b8769dfe0d925c76f6303"} Oct 29 23:00:17.119755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.119735 5736 scope.go:122] "RemoveContainer" containerID="3c8b1a3928a82de2b2c8c9ca64563915a475c59d604b8769dfe0d925c76f6303" Oct 29 23:00:17.120073 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:17.120050 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl_odh-ai-gateway-infra(1e4d6ece-0247-4643-8527-5c94a6085721)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" Oct 29 23:00:17.121302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.121284 5736 generic.go:356] "Generic (PLEG): container finished" podID="7da91507-e921-458e-8985-83b0d34face6" containerID="ad449c9d5565f6e8a3be01934cb556981b9596542c48b4efcc61eb788323eb24" exitCode=60 Oct 29 23:00:17.121385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.121357 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" event={"ID":"7da91507-e921-458e-8985-83b0d34face6","Type":"ContainerDied","Data":"ad449c9d5565f6e8a3be01934cb556981b9596542c48b4efcc61eb788323eb24"} Oct 29 23:00:17.121726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.121704 5736 scope.go:122] "RemoveContainer" containerID="ad449c9d5565f6e8a3be01934cb556981b9596542c48b4efcc61eb788323eb24" Oct 29 23:00:17.122006 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:17.121982 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k_odh-ai-gateway-infra(7da91507-e921-458e-8985-83b0d34face6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" podUID="7da91507-e921-458e-8985-83b0d34face6" Oct 29 23:00:17.134902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.134880 5736 scope.go:122] "RemoveContainer" containerID="fa8bf74442efbea69d049daf2d6216d75c6b0287a4a281d96ae9196d68e43329" Oct 29 23:00:17.153846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.153821 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx"] Oct 29 23:00:17.251562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.251459 5736 scope.go:122] "RemoveContainer" containerID="91fa8fdb574fb9d4d160c22bd0bb3a1ae0ba938e4187d972872eb40733280266" Oct 29 23:00:17.396448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.396404 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:17.559724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.559694 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965a9f37-2d56-48c1-a771-06abfa93ce06-kube-api-access-qnbc8\" (UniqueName: \"kubernetes.io/projected/965a9f37-2d56-48c1-a771-06abfa93ce06-kube-api-access-qnbc8\") pod \"965a9f37-2d56-48c1-a771-06abfa93ce06\" (UID: \"965a9f37-2d56-48c1-a771-06abfa93ce06\") " Oct 29 23:00:17.559893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.559847 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965a9f37-2d56-48c1-a771-06abfa93ce06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a9f37-2d56-48c1-a771-06abfa93ce06-maas-api-service-ca\") pod \"965a9f37-2d56-48c1-a771-06abfa93ce06\" (UID: \"965a9f37-2d56-48c1-a771-06abfa93ce06\") " Oct 29 23:00:17.560122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.560090 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965a9f37-2d56-48c1-a771-06abfa93ce06-maas-api-service-ca" pod "965a9f37-2d56-48c1-a771-06abfa93ce06" (UID: "965a9f37-2d56-48c1-a771-06abfa93ce06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:17.560655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.560636 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965a9f37-2d56-48c1-a771-06abfa93ce06-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.561881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.561856 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965a9f37-2d56-48c1-a771-06abfa93ce06-kube-api-access-qnbc8" pod "965a9f37-2d56-48c1-a771-06abfa93ce06" (UID: "965a9f37-2d56-48c1-a771-06abfa93ce06"). InnerVolumeSpecName "kube-api-access-qnbc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:17.662367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:17.662338 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnbc8\" (UniqueName: \"kubernetes.io/projected/965a9f37-2d56-48c1-a771-06abfa93ce06-kube-api-access-qnbc8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.073543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.073514 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:00:18.170635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.169408 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv"] Oct 29 23:00:18.190457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.190424 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" Oct 29 23:00:18.190705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.190428 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx" event={"ID":"965a9f37-2d56-48c1-a771-06abfa93ce06","Type":"ContainerDied","Data":"f9000a0bd8d59ea4bc9cc702757cf2116156a7e714cbf09f62d4af304f2223fa"} Oct 29 23:00:18.190705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.190640 5736 scope.go:122] "RemoveContainer" containerID="dcb303295db6f31d1bb8843a788a82b2b71a1bfbea5a5fe08df157030496e294" Oct 29 23:00:18.206923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.206873 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7"] Oct 29 23:00:18.217504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.217467 5736 generic.go:356] "Generic (PLEG): container finished" podID="b57724d2-cadd-4ae7-923e-ccb807500264" containerID="230010e160ff8d1fcc7c779844cb56faa0c2a7086e1be005f900ef0db123e242" exitCode=60 Oct 29 23:00:18.217791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.217759 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" event={"ID":"b57724d2-cadd-4ae7-923e-ccb807500264","Type":"ContainerDied","Data":"230010e160ff8d1fcc7c779844cb56faa0c2a7086e1be005f900ef0db123e242"} Oct 29 23:00:18.221474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.221423 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" event={"ID":"abbd2d03-4855-4909-b72c-0b26ef149fda","Type":"ContainerStarted","Data":"beb94ea2a020178161df9b7a9a2fd989c032fd884f5ee68cef397ee71cdac2e5"} Oct 29 23:00:18.248817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.236354 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" event={"ID":"32e55f73-e312-4276-94d0-97ded0d3184d","Type":"ContainerStarted","Data":"9b5d5e96d5a6aac446c1514eb0ae0e0d62a1bc02f45ab5f407974806407958ee"} Oct 29 23:00:18.248817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.246564 5736 scope.go:122] "RemoveContainer" containerID="b27a92ba4f3dd8f3c48993f6cc2e8688892e207b8aee6664f58c3c7b4053f1c7" Oct 29 23:00:18.252426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.252395 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl"] Oct 29 23:00:18.297962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.297863 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k"] Oct 29 23:00:18.316310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.314970 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx"] Oct 29 23:00:18.323138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.323104 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8b57a2-29888580-f8ktx"] Oct 29 23:00:18.515679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.515651 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:18.567107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.567078 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:18.582451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.582421 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:18.595405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.595375 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:18.692466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.692435 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57724d2-cadd-4ae7-923e-ccb807500264-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57724d2-cadd-4ae7-923e-ccb807500264-maas-api-service-ca\") pod \"b57724d2-cadd-4ae7-923e-ccb807500264\" (UID: \"b57724d2-cadd-4ae7-923e-ccb807500264\") " Oct 29 23:00:18.692678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.692478 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4d6ece-0247-4643-8527-5c94a6085721-kube-api-access-n47g8\" (UniqueName: \"kubernetes.io/projected/1e4d6ece-0247-4643-8527-5c94a6085721-kube-api-access-n47g8\") pod \"1e4d6ece-0247-4643-8527-5c94a6085721\" (UID: \"1e4d6ece-0247-4643-8527-5c94a6085721\") " Oct 29 23:00:18.692678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.692512 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4d6ece-0247-4643-8527-5c94a6085721-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4d6ece-0247-4643-8527-5c94a6085721-maas-api-service-ca\") pod \"1e4d6ece-0247-4643-8527-5c94a6085721\" (UID: \"1e4d6ece-0247-4643-8527-5c94a6085721\") " Oct 29 23:00:18.692678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.692540 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-maas-api-service-ca\") pod \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\" (UID: \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\") " Oct 29 23:00:18.692855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.692701 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da91507-e921-458e-8985-83b0d34face6-kube-api-access-4cmd7\" (UniqueName: \"kubernetes.io/projected/7da91507-e921-458e-8985-83b0d34face6-kube-api-access-4cmd7\") pod \"7da91507-e921-458e-8985-83b0d34face6\" (UID: \"7da91507-e921-458e-8985-83b0d34face6\") " Oct 29 23:00:18.692855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.692745 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57724d2-cadd-4ae7-923e-ccb807500264-kube-api-access-jjw96\" (UniqueName: \"kubernetes.io/projected/b57724d2-cadd-4ae7-923e-ccb807500264-kube-api-access-jjw96\") pod \"b57724d2-cadd-4ae7-923e-ccb807500264\" (UID: \"b57724d2-cadd-4ae7-923e-ccb807500264\") " Oct 29 23:00:18.692855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.692796 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7da91507-e921-458e-8985-83b0d34face6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da91507-e921-458e-8985-83b0d34face6-maas-api-service-ca\") pod \"7da91507-e921-458e-8985-83b0d34face6\" (UID: \"7da91507-e921-458e-8985-83b0d34face6\") " Oct 29 23:00:18.692855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.692823 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-kube-api-access-hmdcw\" (UniqueName: \"kubernetes.io/projected/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-kube-api-access-hmdcw\") pod \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\" (UID: \"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88\") " Oct 29 23:00:18.693080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.692901 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57724d2-cadd-4ae7-923e-ccb807500264-maas-api-service-ca" pod "b57724d2-cadd-4ae7-923e-ccb807500264" (UID: "b57724d2-cadd-4ae7-923e-ccb807500264"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:18.693368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.693328 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da91507-e921-458e-8985-83b0d34face6-maas-api-service-ca" pod "7da91507-e921-458e-8985-83b0d34face6" (UID: "7da91507-e921-458e-8985-83b0d34face6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:18.693471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.693397 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-maas-api-service-ca" pod "32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" (UID: "32961c39-a2fe-4a02-a9e0-b6eeaebe1b88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:18.693559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.693541 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4d6ece-0247-4643-8527-5c94a6085721-maas-api-service-ca" pod "1e4d6ece-0247-4643-8527-5c94a6085721" (UID: "1e4d6ece-0247-4643-8527-5c94a6085721"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:18.693699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.693642 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da91507-e921-458e-8985-83b0d34face6-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.693808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.693793 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57724d2-cadd-4ae7-923e-ccb807500264-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.693894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.693881 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.695494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.695465 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-kube-api-access-hmdcw" pod "32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" (UID: "32961c39-a2fe-4a02-a9e0-b6eeaebe1b88"). InnerVolumeSpecName "kube-api-access-hmdcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:18.695898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.695861 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57724d2-cadd-4ae7-923e-ccb807500264-kube-api-access-jjw96" pod "b57724d2-cadd-4ae7-923e-ccb807500264" (UID: "b57724d2-cadd-4ae7-923e-ccb807500264"). InnerVolumeSpecName "kube-api-access-jjw96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:18.696318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.696290 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4d6ece-0247-4643-8527-5c94a6085721-kube-api-access-n47g8" pod "1e4d6ece-0247-4643-8527-5c94a6085721" (UID: "1e4d6ece-0247-4643-8527-5c94a6085721"). InnerVolumeSpecName "kube-api-access-n47g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:18.696513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.696489 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da91507-e921-458e-8985-83b0d34face6-kube-api-access-4cmd7" pod "7da91507-e921-458e-8985-83b0d34face6" (UID: "7da91507-e921-458e-8985-83b0d34face6"). InnerVolumeSpecName "kube-api-access-4cmd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:18.795942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.795903 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cmd7\" (UniqueName: \"kubernetes.io/projected/7da91507-e921-458e-8985-83b0d34face6-kube-api-access-4cmd7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.795942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.795936 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjw96\" (UniqueName: \"kubernetes.io/projected/b57724d2-cadd-4ae7-923e-ccb807500264-kube-api-access-jjw96\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.795942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.795945 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmdcw\" (UniqueName: \"kubernetes.io/projected/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88-kube-api-access-hmdcw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.796170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.795954 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n47g8\" (UniqueName: \"kubernetes.io/projected/1e4d6ece-0247-4643-8527-5c94a6085721-kube-api-access-n47g8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.796170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.795963 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4d6ece-0247-4643-8527-5c94a6085721-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.997169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:18.997138 5736 scope.go:122] "RemoveContainer" containerID="04c3c0023a67521386e90ee0830bb1b2026c691e57a8213806b8c9e4465ef797" Oct 29 23:00:19.012786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.009544 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" path="/var/lib/kubelet/pods/965a9f37-2d56-48c1-a771-06abfa93ce06/volumes" Oct 29 23:00:19.248026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.247953 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" event={"ID":"b57724d2-cadd-4ae7-923e-ccb807500264","Type":"ContainerDied","Data":"cb7e4e2c9f927a485be7138b9ffe9827ea6551b175c46ab47c4572fb5eb043b9"} Oct 29 23:00:19.248418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.248027 5736 scope.go:122] "RemoveContainer" containerID="230010e160ff8d1fcc7c779844cb56faa0c2a7086e1be005f900ef0db123e242" Oct 29 23:00:19.248418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.247964 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv" Oct 29 23:00:19.250328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.250285 5736 generic.go:356] "Generic (PLEG): container finished" podID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerID="beb94ea2a020178161df9b7a9a2fd989c032fd884f5ee68cef397ee71cdac2e5" exitCode=60 Oct 29 23:00:19.250501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.250410 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" event={"ID":"abbd2d03-4855-4909-b72c-0b26ef149fda","Type":"ContainerDied","Data":"beb94ea2a020178161df9b7a9a2fd989c032fd884f5ee68cef397ee71cdac2e5"} Oct 29 23:00:19.250917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.250900 5736 scope.go:122] "RemoveContainer" containerID="beb94ea2a020178161df9b7a9a2fd989c032fd884f5ee68cef397ee71cdac2e5" Oct 29 23:00:19.251213 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:19.251192 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd_odh-ai-gateway-infra(abbd2d03-4855-4909-b72c-0b26ef149fda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" Oct 29 23:00:19.253888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.253861 5736 generic.go:356] "Generic (PLEG): container finished" podID="32e55f73-e312-4276-94d0-97ded0d3184d" containerID="9b5d5e96d5a6aac446c1514eb0ae0e0d62a1bc02f45ab5f407974806407958ee" exitCode=60 Oct 29 23:00:19.253976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.253933 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" event={"ID":"32e55f73-e312-4276-94d0-97ded0d3184d","Type":"ContainerDied","Data":"9b5d5e96d5a6aac446c1514eb0ae0e0d62a1bc02f45ab5f407974806407958ee"} Oct 29 23:00:19.254329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.254313 5736 scope.go:122] "RemoveContainer" containerID="9b5d5e96d5a6aac446c1514eb0ae0e0d62a1bc02f45ab5f407974806407958ee" Oct 29 23:00:19.254683 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:19.254659 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k_odh-ai-gateway-infra(32e55f73-e312-4276-94d0-97ded0d3184d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" Oct 29 23:00:19.255385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.255368 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" Oct 29 23:00:19.255385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.255375 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7" event={"ID":"32961c39-a2fe-4a02-a9e0-b6eeaebe1b88","Type":"ContainerDied","Data":"a577654d81bbd60cce4eef8c9a5ab3d1cdb5b97005c2db0075870fd9402ee4d5"} Oct 29 23:00:19.262744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.259941 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" event={"ID":"1e4d6ece-0247-4643-8527-5c94a6085721","Type":"ContainerDied","Data":"cab409c81d6e4d98bd727d3691669bcd1eeba571aad07a4d8803ece4e542efcf"} Oct 29 23:00:19.262744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.259959 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl" Oct 29 23:00:19.264761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.264743 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" Oct 29 23:00:19.264876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.264756 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k" event={"ID":"7da91507-e921-458e-8985-83b0d34face6","Type":"ContainerDied","Data":"99c71a4b911f1402e8fa732b92b58c9365d06c0e0027465ad88162276fb5be4c"} Oct 29 23:00:19.265015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.264949 5736 scope.go:122] "RemoveContainer" containerID="75dff49093ca6e8493e09948dd77908964bb74f175615dbf9c349a252c0a83fe" Oct 29 23:00:19.279394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.279377 5736 scope.go:122] "RemoveContainer" containerID="a14a34cb72253a98c75c84048873df7d1b1c8fa3108827116485e1544d31e3c9" Oct 29 23:00:19.293412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.293394 5736 scope.go:122] "RemoveContainer" containerID="de1158d4b5842c413fa9724ce614c3ec31bbbe4f1c3aacf3aec138ed817786a8" Oct 29 23:00:19.303078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.303059 5736 scope.go:122] "RemoveContainer" containerID="3c8b1a3928a82de2b2c8c9ca64563915a475c59d604b8769dfe0d925c76f6303" Oct 29 23:00:19.314603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.314567 5736 scope.go:122] "RemoveContainer" containerID="ad449c9d5565f6e8a3be01934cb556981b9596542c48b4efcc61eb788323eb24" Oct 29 23:00:19.375429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.375398 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv"] Oct 29 23:00:19.380615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.380552 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888580-wq5wv"] Oct 29 23:00:19.415347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.415312 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7"] Oct 29 23:00:19.422952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.422923 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888580-wl4f7"] Oct 29 23:00:19.450686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.450661 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl"] Oct 29 23:00:19.455281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.455251 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888580-stjtl"] Oct 29 23:00:19.483392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.483362 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k"] Oct 29 23:00:19.493981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.493958 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k"] Oct 29 23:00:19.497960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.497937 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888580-knh5k"] Oct 29 23:00:19.517155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.517061 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd"] Oct 29 23:00:19.915396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.915316 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9"] Oct 29 23:00:19.916127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916105 5736 cpu_manager.go:414] "containerMap: removing container" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerName="cleanup" Oct 29 23:00:19.916127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916125 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerName="cleanup" Oct 29 23:00:19.916127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916134 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916139 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916147 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916153 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916165 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7da91507-e921-458e-8985-83b0d34face6" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916170 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da91507-e921-458e-8985-83b0d34face6" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916179 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7da91507-e921-458e-8985-83b0d34face6" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916183 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da91507-e921-458e-8985-83b0d34face6" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916190 5736 cpu_manager.go:414] "containerMap: removing container" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916195 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916202 5736 cpu_manager.go:414] "containerMap: removing container" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916207 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916229 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916234 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916241 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916245 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916258 5736 cpu_manager.go:414] "containerMap: removing container" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916263 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916271 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" containerName="cleanup" Oct 29 23:00:19.916374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916275 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916391 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916398 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916406 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916412 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916418 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916423 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da91507-e921-458e-8985-83b0d34face6" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916429 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da91507-e921-458e-8985-83b0d34face6" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916435 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916439 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916447 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" containerName="cleanup" Oct 29 23:00:19.917297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.916455 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" containerName="cleanup" Oct 29 23:00:19.920383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.920366 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:19.934756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.934729 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-df82ee-serving-cert\"" Oct 29 23:00:19.942166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:19.942130 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9"] Oct 29 23:00:20.014305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.014269 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jtd\" (UniqueName: \"kubernetes.io/projected/7974b8ff-aa29-452f-b67e-1729a1ac9f91-kube-api-access-s5jtd\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.014479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.014404 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7974b8ff-aa29-452f-b67e-1729a1ac9f91-openshift-service-ca\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.014634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.014614 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-maas-api-tls\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.014725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.014642 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-metrics-certs\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.118038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.117992 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-maas-api-tls\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.118212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.118051 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-metrics-certs\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.118266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.118236 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jtd\" (UniqueName: \"kubernetes.io/projected/7974b8ff-aa29-452f-b67e-1729a1ac9f91-kube-api-access-s5jtd\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.118407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.118380 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7974b8ff-aa29-452f-b67e-1729a1ac9f91-openshift-service-ca\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.119451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.119398 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7974b8ff-aa29-452f-b67e-1729a1ac9f91-openshift-service-ca\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.129394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.128952 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-maas-api-tls\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.131913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.131061 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-metrics-certs\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.141053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.141027 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jtd\" (UniqueName: \"kubernetes.io/projected/7974b8ff-aa29-452f-b67e-1729a1ac9f91-kube-api-access-s5jtd\") pod \"maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.237231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.237197 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:20.283261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.283232 5736 generic.go:356] "Generic (PLEG): container finished" podID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerID="bfa9fd4263b4ce96956cbf407707c8e26d5bfffb54d7fcf1ddf40c56d6b836c5" exitCode=60 Oct 29 23:00:20.283611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.283310 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" event={"ID":"00a8dffa-76b8-4abe-8dd4-102c367f5ea5","Type":"ContainerDied","Data":"bfa9fd4263b4ce96956cbf407707c8e26d5bfffb54d7fcf1ddf40c56d6b836c5"} Oct 29 23:00:20.283611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.283384 5736 scope.go:122] "RemoveContainer" containerID="04c3c0023a67521386e90ee0830bb1b2026c691e57a8213806b8c9e4465ef797" Oct 29 23:00:20.283900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.283867 5736 scope.go:122] "RemoveContainer" containerID="bfa9fd4263b4ce96956cbf407707c8e26d5bfffb54d7fcf1ddf40c56d6b836c5" Oct 29 23:00:20.284374 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:20.284344 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-fnq9b_odh-ai-gateway-infra(00a8dffa-76b8-4abe-8dd4-102c367f5ea5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" Oct 29 23:00:20.542466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.542435 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:20.544824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.544802 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:20.554413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.554394 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9"] Oct 29 23:00:20.554556 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:20.554533 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7974b8ff_aa29_452f_b67e_1729a1ac9f91.slice/crio-ad9592543da305538033c8ba59f263cebac5fcada713bd449a3bb5b9105edc0d WatchSource:0}: Error finding container ad9592543da305538033c8ba59f263cebac5fcada713bd449a3bb5b9105edc0d: Status 404 returned error can't find the container with id ad9592543da305538033c8ba59f263cebac5fcada713bd449a3bb5b9105edc0d Oct 29 23:00:20.635098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.635061 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e55f73-e312-4276-94d0-97ded0d3184d-kube-api-access-pfwvv\" (UniqueName: \"kubernetes.io/projected/32e55f73-e312-4276-94d0-97ded0d3184d-kube-api-access-pfwvv\") pod \"32e55f73-e312-4276-94d0-97ded0d3184d\" (UID: \"32e55f73-e312-4276-94d0-97ded0d3184d\") " Oct 29 23:00:20.635255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.635189 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbd2d03-4855-4909-b72c-0b26ef149fda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbd2d03-4855-4909-b72c-0b26ef149fda-maas-api-service-ca\") pod \"abbd2d03-4855-4909-b72c-0b26ef149fda\" (UID: \"abbd2d03-4855-4909-b72c-0b26ef149fda\") " Oct 29 23:00:20.635333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.635318 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e55f73-e312-4276-94d0-97ded0d3184d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e55f73-e312-4276-94d0-97ded0d3184d-maas-api-service-ca\") pod \"32e55f73-e312-4276-94d0-97ded0d3184d\" (UID: \"32e55f73-e312-4276-94d0-97ded0d3184d\") " Oct 29 23:00:20.635413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.635396 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbd2d03-4855-4909-b72c-0b26ef149fda-kube-api-access-qvrbv\" (UniqueName: \"kubernetes.io/projected/abbd2d03-4855-4909-b72c-0b26ef149fda-kube-api-access-qvrbv\") pod \"abbd2d03-4855-4909-b72c-0b26ef149fda\" (UID: \"abbd2d03-4855-4909-b72c-0b26ef149fda\") " Oct 29 23:00:20.635563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.635537 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbd2d03-4855-4909-b72c-0b26ef149fda-maas-api-service-ca" pod "abbd2d03-4855-4909-b72c-0b26ef149fda" (UID: "abbd2d03-4855-4909-b72c-0b26ef149fda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.635839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.635795 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e55f73-e312-4276-94d0-97ded0d3184d-maas-api-service-ca" pod "32e55f73-e312-4276-94d0-97ded0d3184d" (UID: "32e55f73-e312-4276-94d0-97ded0d3184d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.636224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.636201 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbd2d03-4855-4909-b72c-0b26ef149fda-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.636224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.636225 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e55f73-e312-4276-94d0-97ded0d3184d-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.637484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.637463 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbd2d03-4855-4909-b72c-0b26ef149fda-kube-api-access-qvrbv" pod "abbd2d03-4855-4909-b72c-0b26ef149fda" (UID: "abbd2d03-4855-4909-b72c-0b26ef149fda"). InnerVolumeSpecName "kube-api-access-qvrbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.637828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.637808 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e55f73-e312-4276-94d0-97ded0d3184d-kube-api-access-pfwvv" pod "32e55f73-e312-4276-94d0-97ded0d3184d" (UID: "32e55f73-e312-4276-94d0-97ded0d3184d"). InnerVolumeSpecName "kube-api-access-pfwvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.737518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.737483 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfwvv\" (UniqueName: \"kubernetes.io/projected/32e55f73-e312-4276-94d0-97ded0d3184d-kube-api-access-pfwvv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.737518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:20.737516 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvrbv\" (UniqueName: \"kubernetes.io/projected/abbd2d03-4855-4909-b72c-0b26ef149fda-kube-api-access-qvrbv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.022130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.021784 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" path="/var/lib/kubelet/pods/1e4d6ece-0247-4643-8527-5c94a6085721/volumes" Oct 29 23:00:21.023184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.022902 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" path="/var/lib/kubelet/pods/32961c39-a2fe-4a02-a9e0-b6eeaebe1b88/volumes" Oct 29 23:00:21.024311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.023636 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7da91507-e921-458e-8985-83b0d34face6" path="/var/lib/kubelet/pods/7da91507-e921-458e-8985-83b0d34face6/volumes" Oct 29 23:00:21.024743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.024709 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" path="/var/lib/kubelet/pods/b57724d2-cadd-4ae7-923e-ccb807500264/volumes" Oct 29 23:00:21.390236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.390182 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" event={"ID":"7974b8ff-aa29-452f-b67e-1729a1ac9f91","Type":"ContainerStarted","Data":"30e05fe19838356d9717dc07c990cd5c51f8af885770eacd014dfbffb83b3ec0"} Oct 29 23:00:21.392707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.392684 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:21.392895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.392876 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" event={"ID":"7974b8ff-aa29-452f-b67e-1729a1ac9f91","Type":"ContainerStarted","Data":"ad9592543da305538033c8ba59f263cebac5fcada713bd449a3bb5b9105edc0d"} Oct 29 23:00:21.420049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.420019 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" Oct 29 23:00:21.425497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.424906 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" Oct 29 23:00:21.425497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.420194 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd" event={"ID":"abbd2d03-4855-4909-b72c-0b26ef149fda","Type":"ContainerDied","Data":"dfb587d1eb586bf1cbac4d77bf3f43c69a86b21e0469a1f05d7726f1382d8c5b"} Oct 29 23:00:21.425497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.425353 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k" event={"ID":"32e55f73-e312-4276-94d0-97ded0d3184d","Type":"ContainerDied","Data":"03e6c81d2c7ec256c65c5388bb28559aeb47d5f40d1206d49008f08a1ffa6325"} Oct 29 23:00:21.425497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.425397 5736 scope.go:122] "RemoveContainer" containerID="beb94ea2a020178161df9b7a9a2fd989c032fd884f5ee68cef397ee71cdac2e5" Oct 29 23:00:21.434020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.433736 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b"] Oct 29 23:00:21.446269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.446099 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" podStartSLOduration=2.096200832 podStartE2EDuration="2.446081774s" totalImagesPullingTime="349.880942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:21.443574193 +0000 UTC m=+1018.559493995" watchObservedRunningTime="2026-10-29 23:00:21.446081774 +0000 UTC m=+1018.562001380" Oct 29 23:00:21.475787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.475761 5736 scope.go:122] "RemoveContainer" containerID="9b5d5e96d5a6aac446c1514eb0ae0e0d62a1bc02f45ab5f407974806407958ee" Oct 29 23:00:21.549045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.549013 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k"] Oct 29 23:00:21.554244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.553899 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-479c2a-29888580-xtl5k"] Oct 29 23:00:21.628600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.626482 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd"] Oct 29 23:00:21.638629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.633912 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888580-q7xqd"] Oct 29 23:00:21.731074 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.731034 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:21.800266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.800235 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-kube-api-access-tcct7\" (UniqueName: \"kubernetes.io/projected/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-kube-api-access-tcct7\") pod \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\" (UID: \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\") " Oct 29 23:00:21.800465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.800284 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-maas-api-service-ca\") pod \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\" (UID: \"00a8dffa-76b8-4abe-8dd4-102c367f5ea5\") " Oct 29 23:00:21.800951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.800917 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-maas-api-service-ca" pod "00a8dffa-76b8-4abe-8dd4-102c367f5ea5" (UID: "00a8dffa-76b8-4abe-8dd4-102c367f5ea5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:21.808815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.808777 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-kube-api-access-tcct7" pod "00a8dffa-76b8-4abe-8dd4-102c367f5ea5" (UID: "00a8dffa-76b8-4abe-8dd4-102c367f5ea5"). InnerVolumeSpecName "kube-api-access-tcct7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:21.902537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.902504 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcct7\" (UniqueName: \"kubernetes.io/projected/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-kube-api-access-tcct7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.902537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:21.902533 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8dffa-76b8-4abe-8dd4-102c367f5ea5-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:22.435456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:22.435426 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" Oct 29 23:00:22.435950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:22.435473 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b" event={"ID":"00a8dffa-76b8-4abe-8dd4-102c367f5ea5","Type":"ContainerDied","Data":"79453d1e17972c12e54af0b8b414cd0ee1195ccc07c272b3ce8b2396fab20723"} Oct 29 23:00:22.435950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:22.435541 5736 scope.go:122] "RemoveContainer" containerID="bfa9fd4263b4ce96956cbf407707c8e26d5bfffb54d7fcf1ddf40c56d6b836c5" Oct 29 23:00:22.483036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:22.483011 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b"] Oct 29 23:00:22.492704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:22.492677 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-fnq9b"] Oct 29 23:00:23.006047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:23.006008 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" path="/var/lib/kubelet/pods/00a8dffa-76b8-4abe-8dd4-102c367f5ea5/volumes" Oct 29 23:00:23.006489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:23.006473 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" path="/var/lib/kubelet/pods/32e55f73-e312-4276-94d0-97ded0d3184d/volumes" Oct 29 23:00:23.006918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:23.006903 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" path="/var/lib/kubelet/pods/abbd2d03-4855-4909-b72c-0b26ef149fda/volumes" Oct 29 23:00:23.194158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:23.194127 5736 scope.go:122] "RemoveContainer" containerID="9158d28675aac49e173824d0760d36b7c450864d134b9e6d23a279bb58eae063" Oct 29 23:00:23.204405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:23.204379 5736 scope.go:122] "RemoveContainer" containerID="fe412e0ecad3632b6379e2743f022e9ea60d7bcfa53b1b4e58d72304645cdcd5" Oct 29 23:00:23.213820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:23.213800 5736 scope.go:122] "RemoveContainer" containerID="31de1efa31ce48ada8b46234ae429b44c9d51efaf0185ff2b0e5009cae29bb57" Oct 29 23:00:23.223349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:23.223323 5736 scope.go:122] "RemoveContainer" containerID="40c75c0f2af3e233fc4183a6527bc42e75d0cb3baa12c6569275cd55ee0147eb" Oct 29 23:00:23.232727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:23.232708 5736 scope.go:122] "RemoveContainer" containerID="b0deaa057a3fbb7a5b626cc02ac201d1c5a480596bc44a31e80a7e54d431b369" Oct 29 23:00:26.109689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:26.109656 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:00:26.110159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:26.109739 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:00:26.824785 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:26.824741 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:00:27.443153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:27.443119 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:00:29.986806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:29.986769 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7"] Oct 29 23:00:29.991906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:29.991880 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c79a4e-69kc7"] Oct 29 23:00:31.004759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:31.004728 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6ce08e-6d1a-435e-b97a-80bfcd3b5463" path="/var/lib/kubelet/pods/3b6ce08e-6d1a-435e-b97a-80bfcd3b5463/volumes" Oct 29 23:00:34.492067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:34.492030 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w"] Oct 29 23:00:34.501435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:34.501408 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-nm59w"] Oct 29 23:00:35.014947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:35.014907 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5848ab58-08cc-4029-8202-c81f5c9fb186" path="/var/lib/kubelet/pods/5848ab58-08cc-4029-8202-c81f5c9fb186/volumes" Oct 29 23:00:38.130879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.130847 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5"] Oct 29 23:00:38.131503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131488 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7da91507-e921-458e-8985-83b0d34face6" containerName="cleanup" Oct 29 23:00:38.131550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131504 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da91507-e921-458e-8985-83b0d34face6" containerName="cleanup" Oct 29 23:00:38.131550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131517 5736 cpu_manager.go:414] "containerMap: removing container" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerName="cleanup" Oct 29 23:00:38.131550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131523 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerName="cleanup" Oct 29 23:00:38.131550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131532 5736 cpu_manager.go:414] "containerMap: removing container" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" containerName="cleanup" Oct 29 23:00:38.131550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131537 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" containerName="cleanup" Oct 29 23:00:38.131550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131546 5736 cpu_manager.go:414] "containerMap: removing container" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerName="cleanup" Oct 29 23:00:38.131550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131552 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131561 5736 cpu_manager.go:414] "containerMap: removing container" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131567 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131591 5736 cpu_manager.go:414] "containerMap: removing container" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131596 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131614 5736 cpu_manager.go:414] "containerMap: removing container" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131619 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131626 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131630 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4d6ece-0247-4643-8527-5c94a6085721" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131638 5736 cpu_manager.go:414] "containerMap: removing container" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131642 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131651 5736 cpu_manager.go:414] "containerMap: removing container" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131655 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131771 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57724d2-cadd-4ae7-923e-ccb807500264" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131781 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da91507-e921-458e-8985-83b0d34face6" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131786 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="965a9f37-2d56-48c1-a771-06abfa93ce06" containerName="cleanup" Oct 29 23:00:38.131785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131791 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="32961c39-a2fe-4a02-a9e0-b6eeaebe1b88" containerName="cleanup" Oct 29 23:00:38.132242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131797 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" containerName="cleanup" Oct 29 23:00:38.132242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131803 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerName="cleanup" Oct 29 23:00:38.132242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131809 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerName="cleanup" Oct 29 23:00:38.132242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131815 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerName="cleanup" Oct 29 23:00:38.132242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131824 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" containerName="cleanup" Oct 29 23:00:38.132242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131829 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerName="cleanup" Oct 29 23:00:38.132242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.131834 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" containerName="cleanup" Oct 29 23:00:38.137879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.137859 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.155285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.155256 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5"] Oct 29 23:00:38.174026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.173998 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:00:38.183380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.183358 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:00:38.186274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.186250 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.186473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.186451 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.186560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.186539 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qx2\" (UniqueName: \"kubernetes.io/projected/27f49e7a-5f31-4208-8a8c-421230ef121c-kube-api-access-56qx2\") pod \"maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.287677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.287638 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qx2\" (UniqueName: \"kubernetes.io/projected/27f49e7a-5f31-4208-8a8c-421230ef121c-kube-api-access-56qx2\") pod \"maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.287891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.287875 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.288027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.288008 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.288728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.288704 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.290313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.290291 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.308758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.308737 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qx2\" (UniqueName: \"kubernetes.io/projected/27f49e7a-5f31-4208-8a8c-421230ef121c-kube-api-access-56qx2\") pod \"maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.453019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.452951 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:38.759417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:38.759391 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5"] Oct 29 23:00:38.761089 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:00:38.761065 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f49e7a_5f31_4208_8a8c_421230ef121c.slice/crio-08c127c956b74e294f47df05eb2dae2f872886d9dfb491d94ff3dcebf2b698c6 WatchSource:0}: Error finding container 08c127c956b74e294f47df05eb2dae2f872886d9dfb491d94ff3dcebf2b698c6: Status 404 returned error can't find the container with id 08c127c956b74e294f47df05eb2dae2f872886d9dfb491d94ff3dcebf2b698c6 Oct 29 23:00:39.610554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:39.610514 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" event={"ID":"27f49e7a-5f31-4208-8a8c-421230ef121c","Type":"ContainerStarted","Data":"44af1f18ec3b5f85633a8d930908b2fa2042c3743b895f8edc861450f8d82dc2"} Oct 29 23:00:39.611062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:39.610573 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" event={"ID":"27f49e7a-5f31-4208-8a8c-421230ef121c","Type":"ContainerStarted","Data":"08c127c956b74e294f47df05eb2dae2f872886d9dfb491d94ff3dcebf2b698c6"} Oct 29 23:00:39.649863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:39.649802 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" podStartSLOduration=1.649783389 podStartE2EDuration="1.649783389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:39.64623065 +0000 UTC m=+1036.762150257" watchObservedRunningTime="2026-10-29 23:00:39.649783389 +0000 UTC m=+1036.765702998" Oct 29 23:00:39.838067 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:39.838022 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f49e7a_5f31_4208_8a8c_421230ef121c.slice/crio-conmon-44af1f18ec3b5f85633a8d930908b2fa2042c3743b895f8edc861450f8d82dc2.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:00:40.628609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:40.628557 5736 generic.go:356] "Generic (PLEG): container finished" podID="27f49e7a-5f31-4208-8a8c-421230ef121c" containerID="44af1f18ec3b5f85633a8d930908b2fa2042c3743b895f8edc861450f8d82dc2" exitCode=0 Oct 29 23:00:40.629194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:40.629148 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" event={"ID":"27f49e7a-5f31-4208-8a8c-421230ef121c","Type":"ContainerDied","Data":"44af1f18ec3b5f85633a8d930908b2fa2042c3743b895f8edc861450f8d82dc2"} Oct 29 23:00:42.174979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.174507 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:42.223607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.218157 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-xks-ca\") pod \"27f49e7a-5f31-4208-8a8c-421230ef121c\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " Oct 29 23:00:42.223607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.220790 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f49e7a-5f31-4208-8a8c-421230ef121c-kube-api-access-56qx2\" (UniqueName: \"kubernetes.io/projected/27f49e7a-5f31-4208-8a8c-421230ef121c-kube-api-access-56qx2\") pod \"27f49e7a-5f31-4208-8a8c-421230ef121c\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " Oct 29 23:00:42.223607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.220948 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-openshift-ca\") pod \"27f49e7a-5f31-4208-8a8c-421230ef121c\" (UID: \"27f49e7a-5f31-4208-8a8c-421230ef121c\") " Oct 29 23:00:42.223607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.222494 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-openshift-ca" pod "27f49e7a-5f31-4208-8a8c-421230ef121c" (UID: "27f49e7a-5f31-4208-8a8c-421230ef121c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:42.250608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.248429 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-xks-ca" pod "27f49e7a-5f31-4208-8a8c-421230ef121c" (UID: "27f49e7a-5f31-4208-8a8c-421230ef121c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:42.328371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.328280 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f49e7a-5f31-4208-8a8c-421230ef121c-kube-api-access-56qx2" pod "27f49e7a-5f31-4208-8a8c-421230ef121c" (UID: "27f49e7a-5f31-4208-8a8c-421230ef121c"). InnerVolumeSpecName "kube-api-access-56qx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:42.420653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.420617 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:42.420872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.420856 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56qx2\" (UniqueName: \"kubernetes.io/projected/27f49e7a-5f31-4208-8a8c-421230ef121c-kube-api-access-56qx2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:42.420993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.420977 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f49e7a-5f31-4208-8a8c-421230ef121c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:42.741709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.741650 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" event={"ID":"27f49e7a-5f31-4208-8a8c-421230ef121c","Type":"ContainerDied","Data":"08c127c956b74e294f47df05eb2dae2f872886d9dfb491d94ff3dcebf2b698c6"} Oct 29 23:00:42.742424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.742397 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08c127c956b74e294f47df05eb2dae2f872886d9dfb491d94ff3dcebf2b698c6" Oct 29 23:00:42.742688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.742670 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5" Oct 29 23:00:42.842601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:42.834993 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:43.048244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.048204 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 23:00:43.049397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.048427 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:43.049625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.048462 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 29 23:00:43.217851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.217811 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:00:43.218439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.218404 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:43.219046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.219022 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:00:43.219225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.219204 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:43.357880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.357753 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 29 23:00:43.397207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.397129 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:43.540826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.540785 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92"] Oct 29 23:00:43.732617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.732552 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:43.902717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:43.902669 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:44.396240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:44.394664 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:44.426961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:44.425532 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:44.426961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:44.426210 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:00:44.436966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:44.436928 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:44.438917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:44.438892 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:00:44.466614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:44.466566 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:00:44.487661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:44.487630 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:00:48.542283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.542220 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" podUID="357963fa-2969-45f8-b032-923cf8ad1003" containerName="payload-processing" containerID="cri-o://4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b" gracePeriod=25 Oct 29 23:00:48.867541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.867514 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 23:00:48.909969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.909928 5736 generic.go:356] "Generic (PLEG): container finished" podID="357963fa-2969-45f8-b032-923cf8ad1003" containerID="4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b" exitCode=0 Oct 29 23:00:48.910204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.910110 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" event={"ID":"357963fa-2969-45f8-b032-923cf8ad1003","Type":"ContainerDied","Data":"4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b"} Oct 29 23:00:48.910204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.910162 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" event={"ID":"357963fa-2969-45f8-b032-923cf8ad1003","Type":"ContainerDied","Data":"6902ec1ad15faae2d4a9124cb29a25f72fa9b29df154c0b66f2d86c990ced362"} Oct 29 23:00:48.910204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.910191 5736 scope.go:122] "RemoveContainer" containerID="4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b" Oct 29 23:00:48.910498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.910321 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92" Oct 29 23:00:48.925507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.925484 5736 scope.go:122] "RemoveContainer" containerID="4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b" Oct 29 23:00:48.925831 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:48.925806 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b\": container with ID starting with 4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b not found: ID does not exist" containerID="4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b" Oct 29 23:00:48.925974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.925949 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b"} err="failed to get container status \"4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b\": rpc error: code = NotFound desc = could not find container \"4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b\": container with ID starting with 4234b97d57d61793bebf64a0b6e8028b0401429d946c4ccbae2378630b962c8b not found: ID does not exist" Oct 29 23:00:48.960020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.959987 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357963fa-2969-45f8-b032-923cf8ad1003-kube-api-access-gg95d\" (UniqueName: \"kubernetes.io/projected/357963fa-2969-45f8-b032-923cf8ad1003-kube-api-access-gg95d\") pod \"357963fa-2969-45f8-b032-923cf8ad1003\" (UID: \"357963fa-2969-45f8-b032-923cf8ad1003\") " Oct 29 23:00:48.960179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.960162 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357963fa-2969-45f8-b032-923cf8ad1003-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357963fa-2969-45f8-b032-923cf8ad1003-plugins-config-volume\") pod \"357963fa-2969-45f8-b032-923cf8ad1003\" (UID: \"357963fa-2969-45f8-b032-923cf8ad1003\") " Oct 29 23:00:48.960630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.960597 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357963fa-2969-45f8-b032-923cf8ad1003-plugins-config-volume" pod "357963fa-2969-45f8-b032-923cf8ad1003" (UID: "357963fa-2969-45f8-b032-923cf8ad1003"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:48.960962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.960939 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357963fa-2969-45f8-b032-923cf8ad1003-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:48.962459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:48.962392 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357963fa-2969-45f8-b032-923cf8ad1003-kube-api-access-gg95d" pod "357963fa-2969-45f8-b032-923cf8ad1003" (UID: "357963fa-2969-45f8-b032-923cf8ad1003"). InnerVolumeSpecName "kube-api-access-gg95d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:49.062686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:49.062643 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg95d\" (UniqueName: \"kubernetes.io/projected/357963fa-2969-45f8-b032-923cf8ad1003-kube-api-access-gg95d\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:00:49.248238 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:49.248204 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92"] Oct 29 23:00:49.259482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:49.259448 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-475876-57b6d6787d-nkf92"] Oct 29 23:00:51.006110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:00:51.006072 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357963fa-2969-45f8-b032-923cf8ad1003" path="/var/lib/kubelet/pods/357963fa-2969-45f8-b032-923cf8ad1003/volumes" Oct 29 23:00:56.835179 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:00:56.835143 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:01:07.108835 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.108801 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp"] Oct 29 23:01:07.109494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109476 5736 cpu_manager.go:414] "containerMap: removing container" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerName="cleanup" Oct 29 23:01:07.109494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109495 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerName="cleanup" Oct 29 23:01:07.109601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109511 5736 cpu_manager.go:414] "containerMap: removing container" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" containerName="cleanup" Oct 29 23:01:07.109601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109516 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e55f73-e312-4276-94d0-97ded0d3184d" containerName="cleanup" Oct 29 23:01:07.109601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109548 5736 cpu_manager.go:414] "containerMap: removing container" podUID="357963fa-2969-45f8-b032-923cf8ad1003" containerName="payload-processing" Oct 29 23:01:07.109601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109554 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357963fa-2969-45f8-b032-923cf8ad1003" containerName="payload-processing" Oct 29 23:01:07.109601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109560 5736 cpu_manager.go:414] "containerMap: removing container" podUID="27f49e7a-5f31-4208-8a8c-421230ef121c" containerName="revoke-keys" Oct 29 23:01:07.109601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109564 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f49e7a-5f31-4208-8a8c-421230ef121c" containerName="revoke-keys" Oct 29 23:01:07.109601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109595 5736 cpu_manager.go:414] "containerMap: removing container" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerName="cleanup" Oct 29 23:01:07.109601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109600 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerName="cleanup" Oct 29 23:01:07.109849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109708 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="357963fa-2969-45f8-b032-923cf8ad1003" containerName="payload-processing" Oct 29 23:01:07.109849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109717 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f49e7a-5f31-4208-8a8c-421230ef121c" containerName="revoke-keys" Oct 29 23:01:07.109849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109723 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a8dffa-76b8-4abe-8dd4-102c367f5ea5" containerName="cleanup" Oct 29 23:01:07.109849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.109732 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbd2d03-4855-4909-b72c-0b26ef149fda" containerName="cleanup" Oct 29 23:01:07.114299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.114280 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.132779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.132754 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-660540-openshift-default-dockercfg-qw84q\"" Oct 29 23:01:07.146912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.146884 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp"] Oct 29 23:01:07.232503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.232472 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9279\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-kube-api-access-k9279\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.232674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.232516 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.232674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.232613 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.232764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.232684 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.232764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.232750 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.232861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.232848 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.232906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.232895 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.232947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.232938 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.233043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.233022 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.333797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.333772 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.333938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.333819 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.333938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.333858 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334048 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334135 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9279\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-kube-api-access-k9279\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334182 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334197 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334234 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334259 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334323 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334376 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334505 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334764 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.334881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.334807 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.336436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.336413 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.336517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.336414 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.355441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.355418 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.356494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.356469 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9279\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-kube-api-access-k9279\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.434950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.434892 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:07.767188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.767164 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp"] Oct 29 23:01:07.768170 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:01:07.768141 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811433d3_39ab_4d7f_9c9e_baa41b8b025b.slice/crio-bf84ea9109640b6c9e21c3814fae7bba8904c76568818ebcce675008ccd9f941 WatchSource:0}: Error finding container bf84ea9109640b6c9e21c3814fae7bba8904c76568818ebcce675008ccd9f941: Status 404 returned error can't find the container with id bf84ea9109640b6c9e21c3814fae7bba8904c76568818ebcce675008ccd9f941 Oct 29 23:01:07.770820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.770783 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:07.770901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.770849 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:07.770901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:07.770881 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:08.152901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:08.152792 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" event={"ID":"811433d3-39ab-4d7f-9c9e-baa41b8b025b","Type":"ContainerStarted","Data":"917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9"} Oct 29 23:01:08.152901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:08.152860 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" event={"ID":"811433d3-39ab-4d7f-9c9e-baa41b8b025b","Type":"ContainerStarted","Data":"bf84ea9109640b6c9e21c3814fae7bba8904c76568818ebcce675008ccd9f941"} Oct 29 23:01:08.198409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:08.198188 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" podStartSLOduration=1.198168946 podStartE2EDuration="1.198168946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:08.195283039 +0000 UTC m=+1065.311202646" watchObservedRunningTime="2026-10-29 23:01:08.198168946 +0000 UTC m=+1065.314088557" Oct 29 23:01:08.461465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:08.460961 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:08.463086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:08.463057 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 23:01:08.463243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:08.463104 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 23:01:09.423373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.423338 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9"] Oct 29 23:01:09.428022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.428003 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:09.445611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.444921 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7c486c\"" Oct 29 23:01:09.458943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.458918 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7c486c-dockercfg-tzw5x\"" Oct 29 23:01:09.459427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.459222 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 23:01:09.459566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.459537 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 23:01:09.466386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.466360 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9"] Oct 29 23:01:09.594751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.589729 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rld\" (UniqueName: \"kubernetes.io/projected/ae4c0a47-9162-4e33-af06-4de03e575b45-kube-api-access-w6rld\") pod \"payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9\" (UID: \"ae4c0a47-9162-4e33-af06-4de03e575b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:09.594751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.589783 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4c0a47-9162-4e33-af06-4de03e575b45-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9\" (UID: \"ae4c0a47-9162-4e33-af06-4de03e575b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:09.610733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.610672 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7"] Oct 29 23:01:09.657274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.644775 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:09.672824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.658707 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7"] Oct 29 23:01:09.781846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.764701 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rld\" (UniqueName: \"kubernetes.io/projected/ae4c0a47-9162-4e33-af06-4de03e575b45-kube-api-access-w6rld\") pod \"payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9\" (UID: \"ae4c0a47-9162-4e33-af06-4de03e575b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:09.781846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.764750 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4c0a47-9162-4e33-af06-4de03e575b45-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9\" (UID: \"ae4c0a47-9162-4e33-af06-4de03e575b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:09.781846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.764872 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pzqz\" (UniqueName: \"kubernetes.io/projected/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-kube-api-access-8pzqz\") pod \"payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7\" (UID: \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:09.781846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.764920 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7\" (UID: \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:09.781846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.766600 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4c0a47-9162-4e33-af06-4de03e575b45-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9\" (UID: \"ae4c0a47-9162-4e33-af06-4de03e575b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:09.830606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.830256 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rld\" (UniqueName: \"kubernetes.io/projected/ae4c0a47-9162-4e33-af06-4de03e575b45-kube-api-access-w6rld\") pod \"payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9\" (UID: \"ae4c0a47-9162-4e33-af06-4de03e575b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:09.896467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.896425 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzqz\" (UniqueName: \"kubernetes.io/projected/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-kube-api-access-8pzqz\") pod \"payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7\" (UID: \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:09.896682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.896494 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7\" (UID: \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:09.902653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.898035 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7\" (UID: \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:09.926113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:09.926087 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzqz\" (UniqueName: \"kubernetes.io/projected/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-kube-api-access-8pzqz\") pod \"payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7\" (UID: \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:10.079837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:10.079751 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:10.199099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:10.196863 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:10.415853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:10.414777 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9"] Oct 29 23:01:10.426963 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:01:10.424019 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4c0a47_9162_4e33_af06_4de03e575b45.slice/crio-70a717cb09345fb5d33c0f035c897e60405f2fcbaa1288bc834f0c28fa3e6f2a WatchSource:0}: Error finding container 70a717cb09345fb5d33c0f035c897e60405f2fcbaa1288bc834f0c28fa3e6f2a: Status 404 returned error can't find the container with id 70a717cb09345fb5d33c0f035c897e60405f2fcbaa1288bc834f0c28fa3e6f2a Oct 29 23:01:10.460773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:10.460735 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 23:01:10.460947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:10.460797 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 23:01:10.607420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:10.606574 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7"] Oct 29 23:01:11.201260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:11.201217 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" event={"ID":"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b","Type":"ContainerStarted","Data":"1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898"} Oct 29 23:01:11.201441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:11.201276 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" event={"ID":"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b","Type":"ContainerStarted","Data":"d1bc60a7fc2e65156504428ec937bcee4595d3b394a06a703a6d9127acf6cc45"} Oct 29 23:01:11.201441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:11.201323 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:11.203454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:11.203425 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" event={"ID":"ae4c0a47-9162-4e33-af06-4de03e575b45","Type":"ContainerStarted","Data":"cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a"} Oct 29 23:01:11.203621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:11.203479 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" event={"ID":"ae4c0a47-9162-4e33-af06-4de03e575b45","Type":"ContainerStarted","Data":"70a717cb09345fb5d33c0f035c897e60405f2fcbaa1288bc834f0c28fa3e6f2a"} Oct 29 23:01:11.203621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:11.203521 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:11.243898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:11.241428 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" podStartSLOduration=2.241404114 podStartE2EDuration="2.241404114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:11.239872503 +0000 UTC m=+1068.355792142" watchObservedRunningTime="2026-10-29 23:01:11.241404114 +0000 UTC m=+1068.357323720" Oct 29 23:01:11.443607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:11.436921 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 23:01:11.443607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:11.436982 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 23:01:12.444603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:12.437939 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 23:01:12.444603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:12.438001 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 23:01:13.455982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:13.455944 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 23:01:13.456368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:13.456007 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 23:01:14.440179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:14.440145 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:14.440625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:14.440608 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:14.441364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:14.441342 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:01:14.504072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:14.504019 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" podStartSLOduration=5.504005809 podStartE2EDuration="5.504005809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:11.2888901 +0000 UTC m=+1068.404809706" watchObservedRunningTime="2026-10-29 23:01:14.504005809 +0000 UTC m=+1071.619925415" Oct 29 23:01:18.821349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:18.821302 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955"] Oct 29 23:01:18.827289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:18.827263 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:18.846227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:18.846201 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955"] Oct 29 23:01:18.852098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:18.852074 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ec4f6d-serving-cert\"" Oct 29 23:01:18.918676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:18.918628 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-metrics-certs\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:18.918880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:18.918775 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5246614-6bd8-496d-b7e3-34a5ff531e5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:18.918880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:18.918845 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52z7\" (UniqueName: \"kubernetes.io/projected/d5246614-6bd8-496d-b7e3-34a5ff531e5e-kube-api-access-t52z7\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:18.918975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:18.918958 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.026605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.026099 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.026605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.026405 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-metrics-certs\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.026605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.026492 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5246614-6bd8-496d-b7e3-34a5ff531e5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.026605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.026555 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52z7\" (UniqueName: \"kubernetes.io/projected/d5246614-6bd8-496d-b7e3-34a5ff531e5e-kube-api-access-t52z7\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.027076 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:01:19.027052 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-serving-cert: secret "maas-api-e2e-api-a-ec4f6d-serving-cert" not found Oct 29 23:01:19.027167 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:01:19.027126 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls podName:d5246614-6bd8-496d-b7e3-34a5ff531e5e nodeName:}" failed. No retries permitted until 2026-10-29 23:01:19.527104178 +0000 UTC m=+1076.643023786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls") pod "maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" (UID: "d5246614-6bd8-496d-b7e3-34a5ff531e5e") : secret "maas-api-e2e-api-a-ec4f6d-serving-cert" not found Oct 29 23:01:19.035116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.035083 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5246614-6bd8-496d-b7e3-34a5ff531e5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.047259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.045989 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-metrics-certs\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.051700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.051664 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52z7\" (UniqueName: \"kubernetes.io/projected/d5246614-6bd8-496d-b7e3-34a5ff531e5e-kube-api-access-t52z7\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.535435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.535401 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.537784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.537762 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls\") pod \"maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:19.744388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:19.744355 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:20.136978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:20.136932 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955"] Oct 29 23:01:20.331988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:20.331946 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" event={"ID":"d5246614-6bd8-496d-b7e3-34a5ff531e5e","Type":"ContainerStarted","Data":"299a962f942a8431c7b7142f5c783c54d113a2df349bef4bc84a400e59baa213"} Oct 29 23:01:21.347292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:21.347233 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" event={"ID":"d5246614-6bd8-496d-b7e3-34a5ff531e5e","Type":"ContainerStarted","Data":"daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115"} Oct 29 23:01:21.348147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:21.347329 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:21.393568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:21.393513 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" podStartSLOduration=3.033828278 podStartE2EDuration="3.393498299s" totalImagesPullingTime="359.670021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:21.390657594 +0000 UTC m=+1078.506577200" watchObservedRunningTime="2026-10-29 23:01:21.393498299 +0000 UTC m=+1078.509417942" Oct 29 23:01:22.216616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.216551 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:01:22.216898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.216877 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:01:22.567083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.567002 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2"] Oct 29 23:01:22.571842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.571821 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:22.589266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.589245 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-50734e\"" Oct 29 23:01:22.592840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.592815 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2"] Oct 29 23:01:22.608275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.608242 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29536dc6-b3a1-4aae-97ba-632cb4f2496e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2\" (UID: \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:22.608408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.608388 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5h8v\" (UniqueName: \"kubernetes.io/projected/29536dc6-b3a1-4aae-97ba-632cb4f2496e-kube-api-access-d5h8v\") pod \"payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2\" (UID: \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:22.620677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.620650 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9"] Oct 29 23:01:22.626204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.626184 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:22.646294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.646264 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-50734e-dockercfg-b42kk\"" Oct 29 23:01:22.646460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.646302 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9"] Oct 29 23:01:22.710031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.709992 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5h8v\" (UniqueName: \"kubernetes.io/projected/29536dc6-b3a1-4aae-97ba-632cb4f2496e-kube-api-access-d5h8v\") pod \"payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2\" (UID: \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:22.710266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.710118 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv262\" (UniqueName: \"kubernetes.io/projected/e41baf80-0657-4e19-afe4-f66fd0aa3805-kube-api-access-wv262\") pod \"payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9\" (UID: \"e41baf80-0657-4e19-afe4-f66fd0aa3805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:22.710371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.710340 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29536dc6-b3a1-4aae-97ba-632cb4f2496e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2\" (UID: \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:22.710492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.710375 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41baf80-0657-4e19-afe4-f66fd0aa3805-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9\" (UID: \"e41baf80-0657-4e19-afe4-f66fd0aa3805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:22.724318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.724284 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29536dc6-b3a1-4aae-97ba-632cb4f2496e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2\" (UID: \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:22.796610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.796041 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5h8v\" (UniqueName: \"kubernetes.io/projected/29536dc6-b3a1-4aae-97ba-632cb4f2496e-kube-api-access-d5h8v\") pod \"payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2\" (UID: \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:22.814379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.814183 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv262\" (UniqueName: \"kubernetes.io/projected/e41baf80-0657-4e19-afe4-f66fd0aa3805-kube-api-access-wv262\") pod \"payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9\" (UID: \"e41baf80-0657-4e19-afe4-f66fd0aa3805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:22.814596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.814453 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41baf80-0657-4e19-afe4-f66fd0aa3805-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9\" (UID: \"e41baf80-0657-4e19-afe4-f66fd0aa3805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:22.815491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.815465 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41baf80-0657-4e19-afe4-f66fd0aa3805-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9\" (UID: \"e41baf80-0657-4e19-afe4-f66fd0aa3805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:22.842682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.841495 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv262\" (UniqueName: \"kubernetes.io/projected/e41baf80-0657-4e19-afe4-f66fd0aa3805-kube-api-access-wv262\") pod \"payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9\" (UID: \"e41baf80-0657-4e19-afe4-f66fd0aa3805\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:22.887185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.887152 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:22.959533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:22.957210 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:23.224333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:23.224303 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2"] Oct 29 23:01:23.331848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:23.331824 5736 scope.go:122] "RemoveContainer" containerID="4148fafc2e15d877c3e7d116e2ec483d438421bf76c132a39e4ce4c1d5110baf" Oct 29 23:01:23.344402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:23.344371 5736 scope.go:122] "RemoveContainer" containerID="25aa74a411f6e01d6f7dbb46faa506d34fcf6b77cb506a2996b2eb81c5e5fc65" Oct 29 23:01:23.356162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:23.355949 5736 scope.go:122] "RemoveContainer" containerID="c8029e3ba17d33782d196ebe68932f27d89fe2c6312fecb5f25249201c8226be" Oct 29 23:01:23.371135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:23.371103 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" event={"ID":"29536dc6-b3a1-4aae-97ba-632cb4f2496e","Type":"ContainerStarted","Data":"bfff6590d9571e07be9f2384c560456ce47d941f65ec813970b945a74bbed3af"} Oct 29 23:01:23.371251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:23.371174 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:23.371251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:23.371197 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" event={"ID":"29536dc6-b3a1-4aae-97ba-632cb4f2496e","Type":"ContainerStarted","Data":"970eb286c39b35e937a3e440f7c136b793b49c6e9bf8aefccc20334a2b346f7c"} Oct 29 23:01:23.386125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:23.386100 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9"] Oct 29 23:01:23.386984 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:01:23.386960 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41baf80_0657_4e19_afe4_f66fd0aa3805.slice/crio-e8f72bb8b9eb68f6056df2ff3841da1450f23dd7c1203fa7fef76871f70983b1 WatchSource:0}: Error finding container e8f72bb8b9eb68f6056df2ff3841da1450f23dd7c1203fa7fef76871f70983b1: Status 404 returned error can't find the container with id e8f72bb8b9eb68f6056df2ff3841da1450f23dd7c1203fa7fef76871f70983b1 Oct 29 23:01:23.416882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:23.416823 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" podStartSLOduration=1.416810139 podStartE2EDuration="1.416810139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:23.41115565 +0000 UTC m=+1080.527075253" watchObservedRunningTime="2026-10-29 23:01:23.416810139 +0000 UTC m=+1080.532729745" Oct 29 23:01:24.385364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:24.385323 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" event={"ID":"e41baf80-0657-4e19-afe4-f66fd0aa3805","Type":"ContainerStarted","Data":"beaef7b9c5bbe8c004937c81812638e50c3438a2e4f87fb68da02008cea8598b"} Oct 29 23:01:24.385879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:24.385388 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" event={"ID":"e41baf80-0657-4e19-afe4-f66fd0aa3805","Type":"ContainerStarted","Data":"e8f72bb8b9eb68f6056df2ff3841da1450f23dd7c1203fa7fef76871f70983b1"} Oct 29 23:01:24.385879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:24.385601 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:24.431359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:24.431303 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" podStartSLOduration=2.431288985 podStartE2EDuration="2.431288985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:24.428145979 +0000 UTC m=+1081.544065598" watchObservedRunningTime="2026-10-29 23:01:24.431288985 +0000 UTC m=+1081.547208591" Oct 29 23:01:26.825867 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:01:26.825830 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:01:27.363682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:27.363654 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:01:30.503243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.503205 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2"] Oct 29 23:01:30.536171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.536138 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2"] Oct 29 23:01:30.536485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.536465 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.553965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.553786 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:01:30.565816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.565791 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:01:30.663971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.663933 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.664229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.664206 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.664308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.664263 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lvw\" (UniqueName: \"kubernetes.io/projected/572768af-11d7-4c2f-958d-a0570f3bda20-kube-api-access-n5lvw\") pod \"maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.766181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.766094 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.766181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.766146 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lvw\" (UniqueName: \"kubernetes.io/projected/572768af-11d7-4c2f-958d-a0570f3bda20-kube-api-access-n5lvw\") pod \"maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.766389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.766320 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.766991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.766963 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.768573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.768551 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.790003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.789976 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lvw\" (UniqueName: \"kubernetes.io/projected/572768af-11d7-4c2f-958d-a0570f3bda20-kube-api-access-n5lvw\") pod \"maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:30.875045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:30.873757 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:31.236601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:31.236557 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2"] Oct 29 23:01:31.238232 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:01:31.238203 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572768af_11d7_4c2f_958d_a0570f3bda20.slice/crio-a44139aba775fea31ede2da51aa61fd93215f72518f72f183c6b774788253167 WatchSource:0}: Error finding container a44139aba775fea31ede2da51aa61fd93215f72518f72f183c6b774788253167: Status 404 returned error can't find the container with id a44139aba775fea31ede2da51aa61fd93215f72518f72f183c6b774788253167 Oct 29 23:01:31.472425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:31.472383 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5"] Oct 29 23:01:31.476789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:31.476756 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" event={"ID":"572768af-11d7-4c2f-958d-a0570f3bda20","Type":"ContainerStarted","Data":"a44139aba775fea31ede2da51aa61fd93215f72518f72f183c6b774788253167"} Oct 29 23:01:31.477148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:31.477126 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8b57a2-tt4f5"] Oct 29 23:01:32.487043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:32.487010 5736 generic.go:356] "Generic (PLEG): container finished" podID="572768af-11d7-4c2f-958d-a0570f3bda20" containerID="cb5b91bd20fc4e836bd59b9ab519960a818911921b7c652337db8c5e5e4a7963" exitCode=0 Oct 29 23:01:32.487434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:32.487051 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" event={"ID":"572768af-11d7-4c2f-958d-a0570f3bda20","Type":"ContainerDied","Data":"cb5b91bd20fc4e836bd59b9ab519960a818911921b7c652337db8c5e5e4a7963"} Oct 29 23:01:33.010269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.010226 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f49e7a-5f31-4208-8a8c-421230ef121c" path="/var/lib/kubelet/pods/27f49e7a-5f31-4208-8a8c-421230ef121c/volumes" Oct 29 23:01:33.736124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.736097 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:33.837904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.837792 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572768af-11d7-4c2f-958d-a0570f3bda20-kube-api-access-n5lvw\" (UniqueName: \"kubernetes.io/projected/572768af-11d7-4c2f-958d-a0570f3bda20-kube-api-access-n5lvw\") pod \"572768af-11d7-4c2f-958d-a0570f3bda20\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " Oct 29 23:01:33.838263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.838245 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-openshift-ca\") pod \"572768af-11d7-4c2f-958d-a0570f3bda20\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " Oct 29 23:01:33.838430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.838407 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-xks-ca\") pod \"572768af-11d7-4c2f-958d-a0570f3bda20\" (UID: \"572768af-11d7-4c2f-958d-a0570f3bda20\") " Oct 29 23:01:33.839171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.839145 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-openshift-ca" pod "572768af-11d7-4c2f-958d-a0570f3bda20" (UID: "572768af-11d7-4c2f-958d-a0570f3bda20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:33.841853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.841826 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572768af-11d7-4c2f-958d-a0570f3bda20-kube-api-access-n5lvw" pod "572768af-11d7-4c2f-958d-a0570f3bda20" (UID: "572768af-11d7-4c2f-958d-a0570f3bda20"). InnerVolumeSpecName "kube-api-access-n5lvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:33.854001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.853972 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-xks-ca" pod "572768af-11d7-4c2f-958d-a0570f3bda20" (UID: "572768af-11d7-4c2f-958d-a0570f3bda20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:33.940565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.940529 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lvw\" (UniqueName: \"kubernetes.io/projected/572768af-11d7-4c2f-958d-a0570f3bda20-kube-api-access-n5lvw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:01:33.940565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.940563 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:01:33.940565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:33.940573 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572768af-11d7-4c2f-958d-a0570f3bda20-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:01:34.392312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:34.392283 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:01:34.512840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:34.512811 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" Oct 29 23:01:34.513028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:34.512893 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2" event={"ID":"572768af-11d7-4c2f-958d-a0570f3bda20","Type":"ContainerDied","Data":"a44139aba775fea31ede2da51aa61fd93215f72518f72f183c6b774788253167"} Oct 29 23:01:34.513028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:34.512935 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44139aba775fea31ede2da51aa61fd93215f72518f72f183c6b774788253167" Oct 29 23:01:35.415600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:35.415080 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:01:37.076643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:37.071627 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g"] Oct 29 23:01:37.113604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:37.105191 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407efa54-c6s9g"] Oct 29 23:01:38.055603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:38.047571 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5"] Oct 29 23:01:38.079606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:38.073289 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e31446-g47l5"] Oct 29 23:01:39.273602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:39.265236 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac4e702-7212-49e6-a211-b6b0ac285e01" path="/var/lib/kubelet/pods/bac4e702-7212-49e6-a211-b6b0ac285e01/volumes" Oct 29 23:01:39.273602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:39.270561 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a7f082-36b4-40b4-9d1a-f87f38eb2735" path="/var/lib/kubelet/pods/c5a7f082-36b4-40b4-9d1a-f87f38eb2735/volumes" Oct 29 23:01:40.384310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.381036 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv"] Oct 29 23:01:40.384310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.382149 5736 cpu_manager.go:414] "containerMap: removing container" podUID="572768af-11d7-4c2f-958d-a0570f3bda20" containerName="revoke-keys" Oct 29 23:01:40.384310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.382172 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572768af-11d7-4c2f-958d-a0570f3bda20" containerName="revoke-keys" Oct 29 23:01:40.384310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.382442 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="572768af-11d7-4c2f-958d-a0570f3bda20" containerName="revoke-keys" Oct 29 23:01:40.387631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.386893 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.404983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.404955 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-44a4e4-openshift-default-dockercfg-qjfbh\"" Oct 29 23:01:40.424352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.424316 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv"] Oct 29 23:01:40.483452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.483418 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.483695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.483470 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.483695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.483520 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.483695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.483545 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.483695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.483561 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.483865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.483723 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.483865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.483795 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwxzb\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-kube-api-access-dwxzb\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.483929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.483868 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.483929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.483909 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.589964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.589920 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwxzb\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-kube-api-access-dwxzb\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.599840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.598750 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.599840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.598900 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.599840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.599523 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.600174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.600040 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.626781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.600242 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.628504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.627466 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.628504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.627532 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.628504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.627763 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.628504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.628186 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.628504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.601529 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.628504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.628441 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.628504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.600645 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.629205 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.629184 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.655291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.652119 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.656142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.655955 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.657339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.657320 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.692283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.692248 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwxzb\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-kube-api-access-dwxzb\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:40.725208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:40.724934 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:41.081106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:41.081068 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q"] Oct 29 23:01:41.081306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:41.081129 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec2f699-jt28q"] Oct 29 23:01:41.255252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:41.255222 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv"] Oct 29 23:01:41.256679 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:01:41.256650 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9be3fc_62e6_4663_abff_5e89dcbfcf46.slice/crio-a4b70ffc88098c3247602053e3282c22a0186b23ba7e5b2c77c3734a8d68735f WatchSource:0}: Error finding container a4b70ffc88098c3247602053e3282c22a0186b23ba7e5b2c77c3734a8d68735f: Status 404 returned error can't find the container with id a4b70ffc88098c3247602053e3282c22a0186b23ba7e5b2c77c3734a8d68735f Oct 29 23:01:41.259858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:41.259819 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:41.259967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:41.259896 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:41.259967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:41.259956 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:41.850172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:41.850129 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" event={"ID":"fd9be3fc-62e6-4663-abff-5e89dcbfcf46","Type":"ContainerStarted","Data":"5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266"} Oct 29 23:01:41.850793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:41.850768 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" event={"ID":"fd9be3fc-62e6-4663-abff-5e89dcbfcf46","Type":"ContainerStarted","Data":"a4b70ffc88098c3247602053e3282c22a0186b23ba7e5b2c77c3734a8d68735f"} Oct 29 23:01:41.902748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:41.902684 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" podStartSLOduration=1.9026647429999999 podStartE2EDuration="1.902664743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:41.896596436 +0000 UTC m=+1099.012516035" watchObservedRunningTime="2026-10-29 23:01:41.902664743 +0000 UTC m=+1099.018584350" Oct 29 23:01:42.725688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:42.725644 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:42.732101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:42.732060 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:42.860603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:42.860415 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:42.861558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:42.861525 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:01:43.010456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:43.010384 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6006b7-2f00-4764-8b9f-20f8e5878211" path="/var/lib/kubelet/pods/3f6006b7-2f00-4764-8b9f-20f8e5878211/volumes" Oct 29 23:01:44.190859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.190819 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6"] Oct 29 23:01:44.198000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.197971 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:44.221420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.221388 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6"] Oct 29 23:01:44.226007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.225912 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-10b990\"" Oct 29 23:01:44.226172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.226083 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-10b990-dockercfg-mfx6c\"" Oct 29 23:01:44.226925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.226895 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f2c51-167f-4efa-a496-c30f745f6e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6\" (UID: \"f90f2c51-167f-4efa-a496-c30f745f6e06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:44.227100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.227080 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvmjt\" (UniqueName: \"kubernetes.io/projected/f90f2c51-167f-4efa-a496-c30f745f6e06-kube-api-access-kvmjt\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6\" (UID: \"f90f2c51-167f-4efa-a496-c30f745f6e06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:44.260523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.260489 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps"] Oct 29 23:01:44.267724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.267697 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:44.303696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.303662 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps"] Oct 29 23:01:44.564375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.564321 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvmjt\" (UniqueName: \"kubernetes.io/projected/f90f2c51-167f-4efa-a496-c30f745f6e06-kube-api-access-kvmjt\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6\" (UID: \"f90f2c51-167f-4efa-a496-c30f745f6e06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:44.564850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.564818 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r66j\" (UniqueName: \"kubernetes.io/projected/39a22f14-1c4d-413c-988a-775d5a72b7d8-kube-api-access-9r66j\") pod \"payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps\" (UID: \"39a22f14-1c4d-413c-988a-775d5a72b7d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:44.565139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.565105 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f2c51-167f-4efa-a496-c30f745f6e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6\" (UID: \"f90f2c51-167f-4efa-a496-c30f745f6e06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:44.602980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.602068 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a22f14-1c4d-413c-988a-775d5a72b7d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps\" (UID: \"39a22f14-1c4d-413c-988a-775d5a72b7d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:44.613702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.613441 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f2c51-167f-4efa-a496-c30f745f6e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6\" (UID: \"f90f2c51-167f-4efa-a496-c30f745f6e06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:44.643398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.643366 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvmjt\" (UniqueName: \"kubernetes.io/projected/f90f2c51-167f-4efa-a496-c30f745f6e06-kube-api-access-kvmjt\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6\" (UID: \"f90f2c51-167f-4efa-a496-c30f745f6e06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:44.704232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.704187 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r66j\" (UniqueName: \"kubernetes.io/projected/39a22f14-1c4d-413c-988a-775d5a72b7d8-kube-api-access-9r66j\") pod \"payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps\" (UID: \"39a22f14-1c4d-413c-988a-775d5a72b7d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:44.704409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.704350 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a22f14-1c4d-413c-988a-775d5a72b7d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps\" (UID: \"39a22f14-1c4d-413c-988a-775d5a72b7d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:44.705140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.705110 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a22f14-1c4d-413c-988a-775d5a72b7d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps\" (UID: \"39a22f14-1c4d-413c-988a-775d5a72b7d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:44.728450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.728415 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r66j\" (UniqueName: \"kubernetes.io/projected/39a22f14-1c4d-413c-988a-775d5a72b7d8-kube-api-access-9r66j\") pod \"payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps\" (UID: \"39a22f14-1c4d-413c-988a-775d5a72b7d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:44.818780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.818659 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:44.917237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:44.917193 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:45.370924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.370890 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps"] Oct 29 23:01:45.371484 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:01:45.371461 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a22f14_1c4d_413c_988a_775d5a72b7d8.slice/crio-9233e58f8531464fe7e2f4569979ace57571028b5d546700925e8752089bf12d WatchSource:0}: Error finding container 9233e58f8531464fe7e2f4569979ace57571028b5d546700925e8752089bf12d: Status 404 returned error can't find the container with id 9233e58f8531464fe7e2f4569979ace57571028b5d546700925e8752089bf12d Oct 29 23:01:45.376996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.376970 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6"] Oct 29 23:01:45.393094 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:01:45.393022 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90f2c51_167f_4efa_a496_c30f745f6e06.slice/crio-95c773a924392a616091d6ed08b64b43b720719089b63d6f1a2ecdfc83a98bca WatchSource:0}: Error finding container 95c773a924392a616091d6ed08b64b43b720719089b63d6f1a2ecdfc83a98bca: Status 404 returned error can't find the container with id 95c773a924392a616091d6ed08b64b43b720719089b63d6f1a2ecdfc83a98bca Oct 29 23:01:45.896032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.895994 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" event={"ID":"f90f2c51-167f-4efa-a496-c30f745f6e06","Type":"ContainerStarted","Data":"528542598fb764a201599ce90e7cd6835b424f43ce86b17b9a02072f23fa0c1e"} Oct 29 23:01:45.896254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.896059 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" event={"ID":"f90f2c51-167f-4efa-a496-c30f745f6e06","Type":"ContainerStarted","Data":"95c773a924392a616091d6ed08b64b43b720719089b63d6f1a2ecdfc83a98bca"} Oct 29 23:01:45.896254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.896092 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:45.898195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.898163 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" event={"ID":"39a22f14-1c4d-413c-988a-775d5a72b7d8","Type":"ContainerStarted","Data":"560fb519185bafdd25f8796805ea35015dd2dc4e7b4c12f0aa01da3d5cf66157"} Oct 29 23:01:45.898307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.898220 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" event={"ID":"39a22f14-1c4d-413c-988a-775d5a72b7d8","Type":"ContainerStarted","Data":"9233e58f8531464fe7e2f4569979ace57571028b5d546700925e8752089bf12d"} Oct 29 23:01:45.898307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.898248 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:45.942960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.942900 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" podStartSLOduration=1.942882305 podStartE2EDuration="1.942882305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:45.937566635 +0000 UTC m=+1103.053486241" watchObservedRunningTime="2026-10-29 23:01:45.942882305 +0000 UTC m=+1103.058801912" Oct 29 23:01:45.987204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:45.987145 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" podStartSLOduration=1.9871316700000001 podStartE2EDuration="1.98713167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:45.983554158 +0000 UTC m=+1103.099473774" watchObservedRunningTime="2026-10-29 23:01:45.98713167 +0000 UTC m=+1103.103051275" Oct 29 23:01:46.910758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:46.910718 5736 generic.go:356] "Generic (PLEG): container finished" podID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerID="9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b" exitCode=0 Oct 29 23:01:46.911683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:46.911650 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" event={"ID":"c57fe8cc-c1f6-4f0d-8a0e-f56430402465","Type":"ContainerDied","Data":"9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b"} Oct 29 23:01:46.912159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:46.912135 5736 scope.go:122] "RemoveContainer" containerID="9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b" Oct 29 23:01:46.914366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:46.914335 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:46.915098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:46.914420 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:46.915098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:46.914469 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:47.928281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:47.928243 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" event={"ID":"c57fe8cc-c1f6-4f0d-8a0e-f56430402465","Type":"ContainerStarted","Data":"74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd"} Oct 29 23:01:48.058641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:48.058589 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc"] Oct 29 23:01:48.063091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:48.063061 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19a9815-qqdbc"] Oct 29 23:01:48.465915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:48.465882 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:01:48.466220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:48.466199 5736 patch_prober.go:29] interesting pod/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:48.466281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:48.466245 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 23:01:49.006082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:49.006049 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7feaea65-696f-479e-a204-035b2b100318" path="/var/lib/kubelet/pods/7feaea65-696f-479e-a204-035b2b100318/volumes" Oct 29 23:01:49.466329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:49.466296 5736 patch_prober.go:29] interesting pod/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:49.466508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:49.466348 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 23:01:50.470602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:50.469952 5736 patch_prober.go:29] interesting pod/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:50.470602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:50.470008 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 23:01:51.088611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:51.081743 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz"] Oct 29 23:01:51.088611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:51.088495 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-129f77ae-gkjlz"] Oct 29 23:01:51.466058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:51.466025 5736 patch_prober.go:29] interesting pod/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:51.466211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:51.466090 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 23:01:52.256127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.256089 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs"] Oct 29 23:01:52.263228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.263199 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.268102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.267931 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs"] Oct 29 23:01:52.273450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.273428 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml"] Oct 29 23:01:52.277854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.277318 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.303698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.303671 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml"] Oct 29 23:01:52.395271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.394855 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.395271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.394906 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.395271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395006 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.395271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395032 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qk9\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-kube-api-access-89qk9\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.395271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395058 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdc52\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-kube-api-access-qdc52\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.395271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395138 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.395271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395219 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.395271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395273 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.395657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395404 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.395657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395538 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.395728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395661 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.395807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395786 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.395898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395880 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.395979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.395964 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.396136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.396116 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.396244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.396192 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.396333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.396317 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.396411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.396396 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.465940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.465900 5736 patch_prober.go:29] interesting pod/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 23:01:52.466104 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.465965 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 23:01:52.498421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.498385 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.498614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.498434 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.498614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.498500 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.498614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.498542 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499001 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499006 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499068 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499134 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499244 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499300 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499367 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499427 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499468 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499510 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499639 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499694 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499702 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499723 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499810 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499857 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qk9\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-kube-api-access-89qk9\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499888 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.499989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499905 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdc52\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-kube-api-access-qdc52\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499954 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.499989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.499978 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.500201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.500065 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.500261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.500235 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.500427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.500400 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.500646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.500627 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.501450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.501427 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.502237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.502215 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.502430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.502411 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.503434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.503410 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.523003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.522923 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdc52\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-kube-api-access-qdc52\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.523243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.523219 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.523540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.523519 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.523752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.523734 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qk9\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-kube-api-access-89qk9\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-28frs\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.580549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.580520 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:52.597299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.597272 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:52.968107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.968049 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:52.968267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.968130 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:52.968267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.968172 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:52.968799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.968773 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs"] Oct 29 23:01:52.971120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.971087 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:52.971230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.971160 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:52.971230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.971206 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:52.972198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:52.972174 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml"] Oct 29 23:01:53.007475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:53.007448 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44da8cd4-821a-4139-8c1d-79e4a41179ed" path="/var/lib/kubelet/pods/44da8cd4-821a-4139-8c1d-79e4a41179ed/volumes" Oct 29 23:01:53.051869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:53.050701 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" event={"ID":"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2","Type":"ContainerStarted","Data":"6cd242bc4350de1ec0de12c8f741bc2e1166e69ee15126bdb047027c0a6c7287"} Oct 29 23:01:53.063453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:53.063409 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" event={"ID":"e37cdb27-0846-40e3-b981-4bcf72fbd83c","Type":"ContainerStarted","Data":"09390b959d5b8bcf4f6fe7e0b5dc1a1fc483940aaf4b7ab5e7653d0f9d91f8b6"} Oct 29 23:01:53.505503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:53.505468 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:01:53.507822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:53.507790 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:01:53.508797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:53.508772 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:01:54.087776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.087735 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" event={"ID":"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2","Type":"ContainerStarted","Data":"3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718"} Oct 29 23:01:54.090217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.090178 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" event={"ID":"e37cdb27-0846-40e3-b981-4bcf72fbd83c","Type":"ContainerStarted","Data":"02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032"} Oct 29 23:01:54.174602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.154561 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" podStartSLOduration=2.1545426819999998 podStartE2EDuration="2.154542682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:54.153043733 +0000 UTC m=+1111.268963341" watchObservedRunningTime="2026-10-29 23:01:54.154542682 +0000 UTC m=+1111.270462290" Oct 29 23:01:54.210440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.210384 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" podStartSLOduration=2.210369425 podStartE2EDuration="2.210369425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:54.205964531 +0000 UTC m=+1111.321884136" watchObservedRunningTime="2026-10-29 23:01:54.210369425 +0000 UTC m=+1111.326289030" Oct 29 23:01:54.580905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.580851 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:01:54.582533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.582508 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:01:54.582683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.582653 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 29 23:01:54.598107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.598081 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:01:54.599351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.599333 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 29 23:01:54.599424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:54.599382 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 29 23:01:55.581033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:55.580993 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:01:55.581625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:55.581060 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 29 23:01:55.598128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:55.598096 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 29 23:01:55.598278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:55.598150 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 29 23:01:56.087659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.087615 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs"] Oct 29 23:01:56.090943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.090920 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w"] Oct 29 23:01:56.094904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.094888 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df0510c-mpjfs"] Oct 29 23:01:56.099015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.098658 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181cb988-vtj7w"] Oct 29 23:01:56.584606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.584252 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:01:56.584606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.584307 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 29 23:01:56.597715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.597690 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 29 23:01:56.597846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.597737 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 29 23:01:56.810237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.810202 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7896-701e00"] Oct 29 23:01:56.815749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.815727 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-701e00" Oct 29 23:01:56.829242 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:01:56.829192 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:01:56.855656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.841078 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xdwj4\"" Oct 29 23:01:56.896957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.896915 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-701e00"] Oct 29 23:01:56.920802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.920773 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:01:56.921254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.921236 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:01:56.926632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:56.926509 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgwkb\" (UniqueName: \"kubernetes.io/projected/9cafccce-e699-4d95-8918-6c7468d825ad-kube-api-access-xgwkb\") pod \"test-curl-7896-701e00\" (UID: \"9cafccce-e699-4d95-8918-6c7468d825ad\") " pod="opendatahub/test-curl-7896-701e00" Oct 29 23:01:57.010750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:57.010714 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19297c31-04a0-4acc-9506-c5149687f819" path="/var/lib/kubelet/pods/19297c31-04a0-4acc-9506-c5149687f819/volumes" Oct 29 23:01:57.011359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:57.011325 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60142a70-c4d6-4efd-9ced-6fb6e72fdc23" path="/var/lib/kubelet/pods/60142a70-c4d6-4efd-9ced-6fb6e72fdc23/volumes" Oct 29 23:01:57.028161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:57.028127 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwkb\" (UniqueName: \"kubernetes.io/projected/9cafccce-e699-4d95-8918-6c7468d825ad-kube-api-access-xgwkb\") pod \"test-curl-7896-701e00\" (UID: \"9cafccce-e699-4d95-8918-6c7468d825ad\") " pod="opendatahub/test-curl-7896-701e00" Oct 29 23:01:57.090913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:57.090877 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwkb\" (UniqueName: \"kubernetes.io/projected/9cafccce-e699-4d95-8918-6c7468d825ad-kube-api-access-xgwkb\") pod \"test-curl-7896-701e00\" (UID: \"9cafccce-e699-4d95-8918-6c7468d825ad\") " pod="opendatahub/test-curl-7896-701e00" Oct 29 23:01:57.162464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:57.162375 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-701e00" Oct 29 23:01:57.674879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:57.673764 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 29 23:01:57.674879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:57.673843 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 29 23:01:57.674879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:57.674270 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:01:57.674879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:57.674323 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 29 23:01:58.013185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:58.013158 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7896-701e00"] Oct 29 23:01:58.227244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:58.227199 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-701e00" event={"ID":"9cafccce-e699-4d95-8918-6c7468d825ad","Type":"ContainerStarted","Data":"22208120f9bc41dd970f7d0be28f22153688f73711c2e36fd61b71843197139b"} Oct 29 23:01:58.585308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:58.585220 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:01:58.585308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:58.585278 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 29 23:01:58.598396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:58.598372 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 29 23:01:58.598524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:58.598417 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 29 23:01:59.283605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:59.283181 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7896-701e00" event={"ID":"9cafccce-e699-4d95-8918-6c7468d825ad","Type":"ContainerStarted","Data":"8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698"} Oct 29 23:01:59.389740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:59.388603 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7896-701e00" podStartSLOduration=3.388563224 podStartE2EDuration="3.388563224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:59.342883978 +0000 UTC m=+1116.458803585" watchObservedRunningTime="2026-10-29 23:01:59.388563224 +0000 UTC m=+1116.504482830" Oct 29 23:01:59.582718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:59.582660 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:01:59.582923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:59.582725 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 29 23:01:59.599345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:59.599290 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 29 23:01:59.599551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:01:59.599349 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 29 23:02:00.362790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.362757 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7896-701e00"] Oct 29 23:02:00.367458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.367430 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7896-701e00"] Oct 29 23:02:00.586197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.586161 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:02:00.586634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.586606 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:02:00.587399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.587377 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:02:00.601994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.601974 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:02:00.602263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.602243 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:02:00.602849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.602833 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:02:00.663165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.663099 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp"] Oct 29 23:02:00.663836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.663822 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9cafccce-e699-4d95-8918-6c7468d825ad" containerName="test-curl-7896-701e00" Oct 29 23:02:00.663877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.663838 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cafccce-e699-4d95-8918-6c7468d825ad" containerName="test-curl-7896-701e00" Oct 29 23:02:00.664003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.663984 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cafccce-e699-4d95-8918-6c7468d825ad" containerName="test-curl-7896-701e00" Oct 29 23:02:00.667817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.667801 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:00.714103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.714069 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:02:00.714499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.714468 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:02:00.715401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.715372 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:00.715517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.715434 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:00.723796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.723738 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6kn\" (UniqueName: \"kubernetes.io/projected/ba0cbdb0-09f7-4203-b0be-21771bfbf504-kube-api-access-8m6kn\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:00.733850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.733817 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp"] Oct 29 23:02:00.826712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.826617 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:00.826970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.826920 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:00.827233 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.827210 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6kn\" (UniqueName: \"kubernetes.io/projected/ba0cbdb0-09f7-4203-b0be-21771bfbf504-kube-api-access-8m6kn\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:00.829160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.829133 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:00.834437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.834414 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:00.856219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:00.853204 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6kn\" (UniqueName: \"kubernetes.io/projected/ba0cbdb0-09f7-4203-b0be-21771bfbf504-kube-api-access-8m6kn\") pod \"maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:01.033140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:01.033089 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7"] Oct 29 23:02:01.037295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:01.037266 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d770a527-g7fq7"] Oct 29 23:02:01.040668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:01.040646 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:01.299115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:01.299022 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7896-701e00" podUID="9cafccce-e699-4d95-8918-6c7468d825ad" containerName="test-curl-7896-701e00" containerID="cri-o://8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698" gracePeriod=2 Oct 29 23:02:01.354104 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:02:01.354068 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0cbdb0_09f7_4203_b0be_21771bfbf504.slice/crio-01b9bfa3ed27781ca6eb5b6293bdbfb494d2640e33dda0aa4ce1bb8c1d1d5a9d WatchSource:0}: Error finding container 01b9bfa3ed27781ca6eb5b6293bdbfb494d2640e33dda0aa4ce1bb8c1d1d5a9d: Status 404 returned error can't find the container with id 01b9bfa3ed27781ca6eb5b6293bdbfb494d2640e33dda0aa4ce1bb8c1d1d5a9d Oct 29 23:02:01.354748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:01.354677 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp"] Oct 29 23:02:02.029235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.029199 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn"] Oct 29 23:02:02.033705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.033669 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e19d84-lt2cn"] Oct 29 23:02:02.308890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.308794 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" event={"ID":"ba0cbdb0-09f7-4203-b0be-21771bfbf504","Type":"ContainerStarted","Data":"f3852526518086a0d93387d38ec6c2d48b8a0d7822fbaf4f1ebe7ad7c8ab1a41"} Oct 29 23:02:02.308890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.308852 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" event={"ID":"ba0cbdb0-09f7-4203-b0be-21771bfbf504","Type":"ContainerStarted","Data":"01b9bfa3ed27781ca6eb5b6293bdbfb494d2640e33dda0aa4ce1bb8c1d1d5a9d"} Oct 29 23:02:02.356106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.356035 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" podStartSLOduration=2.35601747 podStartE2EDuration="2.35601747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:02.35563377 +0000 UTC m=+1119.471553375" watchObservedRunningTime="2026-10-29 23:02:02.35601747 +0000 UTC m=+1119.471937077" Oct 29 23:02:02.617630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.617536 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w"] Oct 29 23:02:02.622699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.622674 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.662505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.662471 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d430df-openshift-default-dockercfg-9mfqg\"" Oct 29 23:02:02.675661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.675625 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w"] Oct 29 23:02:02.678833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.678800 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-data\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.678982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.678887 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-envoy\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.678982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.678926 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-credential-socket\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.678982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.678957 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8984147-87bd-4105-989b-e2beb829eef4-istiod-ca-cert\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.679155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.679044 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8984147-87bd-4105-989b-e2beb829eef4-istio-podinfo\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.679155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.679095 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-socket\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.679268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.679211 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-certs\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.679420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.679397 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdqq\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-kube-api-access-tcdqq\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.679683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.679659 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-istio-token\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.782086 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-istio-token\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.782206 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-data\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.782287 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-envoy\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.782326 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-credential-socket\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.782372 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8984147-87bd-4105-989b-e2beb829eef4-istiod-ca-cert\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.782455 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8984147-87bd-4105-989b-e2beb829eef4-istio-podinfo\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.782518 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-socket\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.782666 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-certs\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.782849 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdqq\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-kube-api-access-tcdqq\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.785087 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-data\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.785941 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-credential-socket\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.786608 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8984147-87bd-4105-989b-e2beb829eef4-istiod-ca-cert\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.792681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.791384 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-certs\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.814760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.806491 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-socket\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.828713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.818556 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8984147-87bd-4105-989b-e2beb829eef4-istio-podinfo\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.845602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.844929 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-envoy\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.851378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.847064 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-istio-token\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.851378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.849263 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdqq\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-kube-api-access-tcdqq\") pod \"e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:02.954679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:02.945522 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:03.122856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.121624 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a35a194-d69e-47f2-ac79-464a4a82b202" path="/var/lib/kubelet/pods/3a35a194-d69e-47f2-ac79-464a4a82b202/volumes" Oct 29 23:02:03.124827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.123359 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61e80b5-f3f1-4951-8960-ed0fb44f94a7" path="/var/lib/kubelet/pods/e61e80b5-f3f1-4951-8960-ed0fb44f94a7/volumes" Oct 29 23:02:03.377242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.376686 5736 generic.go:356] "Generic (PLEG): container finished" podID="ba0cbdb0-09f7-4203-b0be-21771bfbf504" containerID="f3852526518086a0d93387d38ec6c2d48b8a0d7822fbaf4f1ebe7ad7c8ab1a41" exitCode=0 Oct 29 23:02:03.377242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.376795 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" event={"ID":"ba0cbdb0-09f7-4203-b0be-21771bfbf504","Type":"ContainerDied","Data":"f3852526518086a0d93387d38ec6c2d48b8a0d7822fbaf4f1ebe7ad7c8ab1a41"} Oct 29 23:02:03.427532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.427495 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w"] Oct 29 23:02:03.440195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.440141 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:03.440424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.440401 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:03.440572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.440554 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:03.602331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.602304 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-701e00" Oct 29 23:02:03.703752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.703719 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cafccce-e699-4d95-8918-6c7468d825ad-kube-api-access-xgwkb\" (UniqueName: \"kubernetes.io/projected/9cafccce-e699-4d95-8918-6c7468d825ad-kube-api-access-xgwkb\") pod \"9cafccce-e699-4d95-8918-6c7468d825ad\" (UID: \"9cafccce-e699-4d95-8918-6c7468d825ad\") " Oct 29 23:02:03.705897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.705862 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cafccce-e699-4d95-8918-6c7468d825ad-kube-api-access-xgwkb" pod "9cafccce-e699-4d95-8918-6c7468d825ad" (UID: "9cafccce-e699-4d95-8918-6c7468d825ad"). InnerVolumeSpecName "kube-api-access-xgwkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:03.806496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:03.806454 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgwkb\" (UniqueName: \"kubernetes.io/projected/9cafccce-e699-4d95-8918-6c7468d825ad-kube-api-access-xgwkb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:04.388308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.388271 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" event={"ID":"e8984147-87bd-4105-989b-e2beb829eef4","Type":"ContainerStarted","Data":"eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7"} Oct 29 23:02:04.388784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.388332 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" event={"ID":"e8984147-87bd-4105-989b-e2beb829eef4","Type":"ContainerStarted","Data":"03e75976ce70192c231f17d7d701c6267218162b91f22baccec0599481faee99"} Oct 29 23:02:04.389851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.389823 5736 generic.go:356] "Generic (PLEG): container finished" podID="9cafccce-e699-4d95-8918-6c7468d825ad" containerID="8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698" exitCode=137 Oct 29 23:02:04.389973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.389860 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7896-701e00" Oct 29 23:02:04.389973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.389913 5736 scope.go:122] "RemoveContainer" containerID="8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698" Oct 29 23:02:04.402368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.402344 5736 scope.go:122] "RemoveContainer" containerID="8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698" Oct 29 23:02:04.402731 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:04.402701 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698\": container with ID starting with 8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698 not found: ID does not exist" containerID="8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698" Oct 29 23:02:04.402821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.402741 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698"} err="failed to get container status \"8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698\": rpc error: code = NotFound desc = could not find container \"8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698\": container with ID starting with 8311cd295715fe0cbac40f821234579a17c3a400f76eeb8400fe6adbacd3f698 not found: ID does not exist" Oct 29 23:02:04.450327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.449617 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" podStartSLOduration=2.449563158 podStartE2EDuration="2.449563158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:04.448346183 +0000 UTC m=+1121.564265817" watchObservedRunningTime="2026-10-29 23:02:04.449563158 +0000 UTC m=+1121.565482766" Oct 29 23:02:04.462458 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:04.462411 5736 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-701e00\" is forbidden: User \"system:node:ip-10-0-143-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-172.ec2.internal' and this object" podUID="9cafccce-e699-4d95-8918-6c7468d825ad" pod="opendatahub/test-curl-7896-701e00" Oct 29 23:02:04.596269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.596225 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:04.620016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.619982 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0cbdb0-09f7-4203-b0be-21771bfbf504-kube-api-access-8m6kn\" (UniqueName: \"kubernetes.io/projected/ba0cbdb0-09f7-4203-b0be-21771bfbf504-kube-api-access-8m6kn\") pod \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " Oct 29 23:02:04.620212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.620162 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-openshift-ca\") pod \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " Oct 29 23:02:04.620352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.620324 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-xks-ca\") pod \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\" (UID: \"ba0cbdb0-09f7-4203-b0be-21771bfbf504\") " Oct 29 23:02:04.620545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.620518 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-openshift-ca" pod "ba0cbdb0-09f7-4203-b0be-21771bfbf504" (UID: "ba0cbdb0-09f7-4203-b0be-21771bfbf504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:04.621312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.621286 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:04.622892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.622854 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-xks-ca" pod "ba0cbdb0-09f7-4203-b0be-21771bfbf504" (UID: "ba0cbdb0-09f7-4203-b0be-21771bfbf504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:04.622892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.622864 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0cbdb0-09f7-4203-b0be-21771bfbf504-kube-api-access-8m6kn" pod "ba0cbdb0-09f7-4203-b0be-21771bfbf504" (UID: "ba0cbdb0-09f7-4203-b0be-21771bfbf504"). InnerVolumeSpecName "kube-api-access-8m6kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:04.650743 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:04.650694 5736 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-701e00\" is forbidden: User \"system:node:ip-10-0-143-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-172.ec2.internal' and this object" podUID="9cafccce-e699-4d95-8918-6c7468d825ad" pod="opendatahub/test-curl-7896-701e00" Oct 29 23:02:04.729706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.729655 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6kn\" (UniqueName: \"kubernetes.io/projected/ba0cbdb0-09f7-4203-b0be-21771bfbf504-kube-api-access-8m6kn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:04.729706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.729697 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0cbdb0-09f7-4203-b0be-21771bfbf504-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:04.959806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.959765 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:04.966714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:04.966689 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:04.980329 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:04.980296 5736 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7896-701e00\" is forbidden: User \"system:node:ip-10-0-143-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-172.ec2.internal' and this object" podUID="9cafccce-e699-4d95-8918-6c7468d825ad" pod="opendatahub/test-curl-7896-701e00" Oct 29 23:02:05.005886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:05.005855 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cafccce-e699-4d95-8918-6c7468d825ad" path="/var/lib/kubelet/pods/9cafccce-e699-4d95-8918-6c7468d825ad/volumes" Oct 29 23:02:05.401294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:05.401200 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" Oct 29 23:02:05.401294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:05.401221 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp" event={"ID":"ba0cbdb0-09f7-4203-b0be-21771bfbf504","Type":"ContainerDied","Data":"01b9bfa3ed27781ca6eb5b6293bdbfb494d2640e33dda0aa4ce1bb8c1d1d5a9d"} Oct 29 23:02:05.401776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:05.401308 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01b9bfa3ed27781ca6eb5b6293bdbfb494d2640e33dda0aa4ce1bb8c1d1d5a9d" Oct 29 23:02:05.402539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:05.402515 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:05.403683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:05.403660 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:02:11.030705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:11.030667 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls"] Oct 29 23:02:11.042014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:11.041984 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86b21d2-fw6ls"] Oct 29 23:02:13.005633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:13.005600 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55cac58-d1b3-43b4-b3a5-6fbba73240b7" path="/var/lib/kubelet/pods/e55cac58-d1b3-43b4-b3a5-6fbba73240b7/volumes" Oct 29 23:02:17.030414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.030376 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j"] Oct 29 23:02:17.042072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.042042 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aca3e648-n8k9j"] Oct 29 23:02:17.685127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.685082 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz"] Oct 29 23:02:17.685950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.685928 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ba0cbdb0-09f7-4203-b0be-21771bfbf504" containerName="revoke-keys" Oct 29 23:02:17.685950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.685950 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0cbdb0-09f7-4203-b0be-21771bfbf504" containerName="revoke-keys" Oct 29 23:02:17.686147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.686134 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0cbdb0-09f7-4203-b0be-21771bfbf504" containerName="revoke-keys" Oct 29 23:02:17.690734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.690712 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.707617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.707510 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-10b990-serving-cert\"" Oct 29 23:02:17.713201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.713171 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.713335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.713288 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5349e38-94a6-483b-a430-57bb302714d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.713479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.713459 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2jz\" (UniqueName: \"kubernetes.io/projected/e5349e38-94a6-483b-a430-57bb302714d5-kube-api-access-hl2jz\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.713557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.713510 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.726146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.726116 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz"] Oct 29 23:02:17.814826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.814788 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5349e38-94a6-483b-a430-57bb302714d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.815013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.814935 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2jz\" (UniqueName: \"kubernetes.io/projected/e5349e38-94a6-483b-a430-57bb302714d5-kube-api-access-hl2jz\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.815013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.814965 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.815152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.815088 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.815214 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:17.815167 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-serving-cert: secret "maas-api-e2e-ipp-b-10b990-serving-cert" not found Oct 29 23:02:17.815274 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:17.815247 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls podName:e5349e38-94a6-483b-a430-57bb302714d5 nodeName:}" failed. No retries permitted until 2026-10-29 23:02:18.315227159 +0000 UTC m=+1135.431146761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls") pod "maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" (UID: "e5349e38-94a6-483b-a430-57bb302714d5") : secret "maas-api-e2e-ipp-b-10b990-serving-cert" not found Oct 29 23:02:17.815543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.815520 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5349e38-94a6-483b-a430-57bb302714d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.817475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.817455 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:17.841021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:17.840988 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2jz\" (UniqueName: \"kubernetes.io/projected/e5349e38-94a6-483b-a430-57bb302714d5-kube-api-access-hl2jz\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:18.323897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:18.323851 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:18.326281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:18.326258 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:18.607657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:18.606567 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:19.084632 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:02:19.083551 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5349e38_94a6_483b_a430_57bb302714d5.slice/crio-1cc9b09ac3b000b069b74286658a0fc12807d501fd63eba1d92f2afd9004de81 WatchSource:0}: Error finding container 1cc9b09ac3b000b069b74286658a0fc12807d501fd63eba1d92f2afd9004de81: Status 404 returned error can't find the container with id 1cc9b09ac3b000b069b74286658a0fc12807d501fd63eba1d92f2afd9004de81 Oct 29 23:02:19.096741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:19.096690 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b9725d-7a83-4ccc-bc79-0f855d4fd875" path="/var/lib/kubelet/pods/43b9725d-7a83-4ccc-bc79-0f855d4fd875/volumes" Oct 29 23:02:19.097269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:19.097245 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz"] Oct 29 23:02:19.533438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:19.533404 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" event={"ID":"e5349e38-94a6-483b-a430-57bb302714d5","Type":"ContainerStarted","Data":"1cc9b09ac3b000b069b74286658a0fc12807d501fd63eba1d92f2afd9004de81"} Oct 29 23:02:20.543449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:20.543412 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" event={"ID":"e5349e38-94a6-483b-a430-57bb302714d5","Type":"ContainerStarted","Data":"502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17"} Oct 29 23:02:20.543862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:20.543472 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:20.587533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:20.587488 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" podStartSLOduration=3.233864126 podStartE2EDuration="3.587476502s" totalImagesPullingTime="353.612376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:20.586729248 +0000 UTC m=+1137.702648856" watchObservedRunningTime="2026-10-29 23:02:20.587476502 +0000 UTC m=+1137.703396108" Oct 29 23:02:22.575498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:22.575464 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2"] Oct 29 23:02:22.586979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:22.586952 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479c2a-jw5b2"] Oct 29 23:02:23.006964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:23.006920 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572768af-11d7-4c2f-958d-a0570f3bda20" path="/var/lib/kubelet/pods/572768af-11d7-4c2f-958d-a0570f3bda20/volumes" Oct 29 23:02:25.101156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:25.101112 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg"] Oct 29 23:02:25.115907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:25.115877 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0a20e-g79zg"] Oct 29 23:02:26.594411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:26.594383 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:02:26.826377 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:26.826337 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:02:27.005592 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:27.005547 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926af2cf-f48c-4361-8457-42b7b2db9ac1" path="/var/lib/kubelet/pods/926af2cf-f48c-4361-8457-42b7b2db9ac1/volumes" Oct 29 23:02:30.768317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.768284 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs"] Oct 29 23:02:30.775682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.775658 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:30.795101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.795076 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs"] Oct 29 23:02:30.800532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.800501 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7jq8\" (UniqueName: \"kubernetes.io/projected/eb843c5d-abb5-4886-8085-2308e71fb2d0-kube-api-access-f7jq8\") pod \"maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:30.807320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.801460 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:02:30.807487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.801481 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:02:30.807572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.807399 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:30.807715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.807630 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:30.909774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.909735 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jq8\" (UniqueName: \"kubernetes.io/projected/eb843c5d-abb5-4886-8085-2308e71fb2d0-kube-api-access-f7jq8\") pod \"maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:30.909986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.909899 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:30.910069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.910021 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:30.910806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.910772 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:30.913945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.913921 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:30.933226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:30.933200 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7jq8\" (UniqueName: \"kubernetes.io/projected/eb843c5d-abb5-4886-8085-2308e71fb2d0-kube-api-access-f7jq8\") pod \"maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:31.108869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.108783 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:31.269215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.269180 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8"] Oct 29 23:02:31.275730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.275702 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.286560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.286531 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8"] Oct 29 23:02:31.317050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.317012 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56t5r\" (UniqueName: \"kubernetes.io/projected/099592c0-0b2d-4ea6-a239-2d98a92e52c0-kube-api-access-56t5r\") pod \"maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.317395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.317371 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.318064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.317902 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.419535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.419452 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56t5r\" (UniqueName: \"kubernetes.io/projected/099592c0-0b2d-4ea6-a239-2d98a92e52c0-kube-api-access-56t5r\") pod \"maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.419754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.419612 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.419754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.419708 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.420332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.420311 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.421942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.421921 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.430568 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:02:31.430532 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb843c5d_abb5_4886_8085_2308e71fb2d0.slice/crio-a040bfcfae9b1dc71919edbaee85f5201c895b2295611fc703c5a492c90f3646 WatchSource:0}: Error finding container a040bfcfae9b1dc71919edbaee85f5201c895b2295611fc703c5a492c90f3646: Status 404 returned error can't find the container with id a040bfcfae9b1dc71919edbaee85f5201c895b2295611fc703c5a492c90f3646 Oct 29 23:02:31.431187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.431162 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs"] Oct 29 23:02:31.442663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.442643 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56t5r\" (UniqueName: \"kubernetes.io/projected/099592c0-0b2d-4ea6-a239-2d98a92e52c0-kube-api-access-56t5r\") pod \"maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.597038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.597004 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:31.698936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.698902 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" event={"ID":"eb843c5d-abb5-4886-8085-2308e71fb2d0","Type":"ContainerStarted","Data":"a040bfcfae9b1dc71919edbaee85f5201c895b2295611fc703c5a492c90f3646"} Oct 29 23:02:31.944438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:31.944271 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8"] Oct 29 23:02:32.712380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:32.712337 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" event={"ID":"099592c0-0b2d-4ea6-a239-2d98a92e52c0","Type":"ContainerStarted","Data":"aff0b072f6a5a80db99b9af6779e33bb4c8e4a8911d2c23aa59c9052516b2dd8"} Oct 29 23:02:32.712549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:32.712443 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" event={"ID":"099592c0-0b2d-4ea6-a239-2d98a92e52c0","Type":"ContainerStarted","Data":"cf8c3f93b21e246d3b6c80ec9f841f0f865ed4f2ec8f47b77bd86459a91eaafa"} Oct 29 23:02:32.714625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:32.714594 5736 generic.go:356] "Generic (PLEG): container finished" podID="eb843c5d-abb5-4886-8085-2308e71fb2d0" containerID="7829bd7de50ae15f74742fd722be0f7566c8e3fa45f1298d0b645f2e15b11e17" exitCode=0 Oct 29 23:02:32.714739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:32.714707 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" event={"ID":"eb843c5d-abb5-4886-8085-2308e71fb2d0","Type":"ContainerDied","Data":"7829bd7de50ae15f74742fd722be0f7566c8e3fa45f1298d0b645f2e15b11e17"} Oct 29 23:02:32.754618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:32.754525 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" podStartSLOduration=1.7544983219999999 podStartE2EDuration="1.754498322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:32.752454958 +0000 UTC m=+1149.868374565" watchObservedRunningTime="2026-10-29 23:02:32.754498322 +0000 UTC m=+1149.870417928" Oct 29 23:02:33.607961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:33.607913 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp"] Oct 29 23:02:33.614290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:33.614259 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79e6c631-kqcnp"] Oct 29 23:02:33.728993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:33.728959 5736 generic.go:356] "Generic (PLEG): container finished" podID="099592c0-0b2d-4ea6-a239-2d98a92e52c0" containerID="aff0b072f6a5a80db99b9af6779e33bb4c8e4a8911d2c23aa59c9052516b2dd8" exitCode=0 Oct 29 23:02:33.729183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:33.729038 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" event={"ID":"099592c0-0b2d-4ea6-a239-2d98a92e52c0","Type":"ContainerDied","Data":"aff0b072f6a5a80db99b9af6779e33bb4c8e4a8911d2c23aa59c9052516b2dd8"} Oct 29 23:02:33.933796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:33.933768 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:34.003522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.003494 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb843c5d-abb5-4886-8085-2308e71fb2d0-kube-api-access-f7jq8\" (UniqueName: \"kubernetes.io/projected/eb843c5d-abb5-4886-8085-2308e71fb2d0-kube-api-access-f7jq8\") pod \"eb843c5d-abb5-4886-8085-2308e71fb2d0\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " Oct 29 23:02:34.003687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.003576 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-openshift-ca\") pod \"eb843c5d-abb5-4886-8085-2308e71fb2d0\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " Oct 29 23:02:34.003687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.003618 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-xks-ca\") pod \"eb843c5d-abb5-4886-8085-2308e71fb2d0\" (UID: \"eb843c5d-abb5-4886-8085-2308e71fb2d0\") " Oct 29 23:02:34.003934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.003908 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-openshift-ca" pod "eb843c5d-abb5-4886-8085-2308e71fb2d0" (UID: "eb843c5d-abb5-4886-8085-2308e71fb2d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:34.004342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.004324 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:34.005591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.005561 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-xks-ca" pod "eb843c5d-abb5-4886-8085-2308e71fb2d0" (UID: "eb843c5d-abb5-4886-8085-2308e71fb2d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:34.005692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.005671 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb843c5d-abb5-4886-8085-2308e71fb2d0-kube-api-access-f7jq8" pod "eb843c5d-abb5-4886-8085-2308e71fb2d0" (UID: "eb843c5d-abb5-4886-8085-2308e71fb2d0"). InnerVolumeSpecName "kube-api-access-f7jq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:34.106340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.106305 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7jq8\" (UniqueName: \"kubernetes.io/projected/eb843c5d-abb5-4886-8085-2308e71fb2d0-kube-api-access-f7jq8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:34.106340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.106343 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb843c5d-abb5-4886-8085-2308e71fb2d0-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:34.741868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.741804 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" Oct 29 23:02:34.741868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.741841 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs" event={"ID":"eb843c5d-abb5-4886-8085-2308e71fb2d0","Type":"ContainerDied","Data":"a040bfcfae9b1dc71919edbaee85f5201c895b2295611fc703c5a492c90f3646"} Oct 29 23:02:34.742487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.741899 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a040bfcfae9b1dc71919edbaee85f5201c895b2295611fc703c5a492c90f3646" Oct 29 23:02:34.926530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:34.926504 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:35.014752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.014677 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0cbdb0-09f7-4203-b0be-21771bfbf504" path="/var/lib/kubelet/pods/ba0cbdb0-09f7-4203-b0be-21771bfbf504/volumes" Oct 29 23:02:35.029698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.029666 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-openshift-ca\") pod \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " Oct 29 23:02:35.029867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.029820 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-xks-ca\") pod \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " Oct 29 23:02:35.030008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.029987 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099592c0-0b2d-4ea6-a239-2d98a92e52c0-kube-api-access-56t5r\" (UniqueName: \"kubernetes.io/projected/099592c0-0b2d-4ea6-a239-2d98a92e52c0-kube-api-access-56t5r\") pod \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\" (UID: \"099592c0-0b2d-4ea6-a239-2d98a92e52c0\") " Oct 29 23:02:35.030127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.030021 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-openshift-ca" pod "099592c0-0b2d-4ea6-a239-2d98a92e52c0" (UID: "099592c0-0b2d-4ea6-a239-2d98a92e52c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:35.031092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.030982 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.032799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.032770 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099592c0-0b2d-4ea6-a239-2d98a92e52c0-kube-api-access-56t5r" pod "099592c0-0b2d-4ea6-a239-2d98a92e52c0" (UID: "099592c0-0b2d-4ea6-a239-2d98a92e52c0"). InnerVolumeSpecName "kube-api-access-56t5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:35.034711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.034687 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-xks-ca" pod "099592c0-0b2d-4ea6-a239-2d98a92e52c0" (UID: "099592c0-0b2d-4ea6-a239-2d98a92e52c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:35.132726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.132695 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/099592c0-0b2d-4ea6-a239-2d98a92e52c0-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.132726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.132724 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56t5r\" (UniqueName: \"kubernetes.io/projected/099592c0-0b2d-4ea6-a239-2d98a92e52c0-kube-api-access-56t5r\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:35.677594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.677476 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5"] Oct 29 23:02:35.678723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.678699 5736 cpu_manager.go:414] "containerMap: removing container" podUID="eb843c5d-abb5-4886-8085-2308e71fb2d0" containerName="revoke-keys" Oct 29 23:02:35.678723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.678721 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb843c5d-abb5-4886-8085-2308e71fb2d0" containerName="revoke-keys" Oct 29 23:02:35.678889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.678743 5736 cpu_manager.go:414] "containerMap: removing container" podUID="099592c0-0b2d-4ea6-a239-2d98a92e52c0" containerName="revoke-keys" Oct 29 23:02:35.678889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.678748 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099592c0-0b2d-4ea6-a239-2d98a92e52c0" containerName="revoke-keys" Oct 29 23:02:35.678951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.678933 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb843c5d-abb5-4886-8085-2308e71fb2d0" containerName="revoke-keys" Oct 29 23:02:35.678951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.678942 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="099592c0-0b2d-4ea6-a239-2d98a92e52c0" containerName="revoke-keys" Oct 29 23:02:35.683154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.683129 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.698436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.698412 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk"] Oct 29 23:02:35.702863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.702845 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.705315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.705297 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ec4f6d-openshift-default-dockercfg-rrkm9\"" Oct 29 23:02:35.716672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.716601 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5"] Oct 29 23:02:35.747092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.747002 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk"] Oct 29 23:02:35.755327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.755101 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" event={"ID":"099592c0-0b2d-4ea6-a239-2d98a92e52c0","Type":"ContainerDied","Data":"cf8c3f93b21e246d3b6c80ec9f841f0f865ed4f2ec8f47b77bd86459a91eaafa"} Oct 29 23:02:35.755327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.755142 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf8c3f93b21e246d3b6c80ec9f841f0f865ed4f2ec8f47b77bd86459a91eaafa" Oct 29 23:02:35.755327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.755155 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8" Oct 29 23:02:35.844388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844357 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.844561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844442 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vslj\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-kube-api-access-2vslj\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.844561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844485 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2eb25d-0a9e-4028-9c70-c8baec507384-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.844650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844568 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.844692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844676 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.844728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844702 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.844786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844774 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.844823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844793 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.844869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844838 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.844963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.844950 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.845024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.845011 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.845092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.845078 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.845139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.845103 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f308b0-4938-435d-ac0d-9630aa87b92b-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.845186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.845150 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.845243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.845197 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.845351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.845321 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.845475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.845397 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmp4w\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-kube-api-access-fmp4w\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.845518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.845476 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.946981 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947047 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.947301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947080 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vslj\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-kube-api-access-2vslj\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947100 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2eb25d-0a9e-4028-9c70-c8baec507384-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.947301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947128 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.947301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947185 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.947301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947212 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947255 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.947301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947276 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947335 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947387 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947421 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947767 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.947828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947793 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947837 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.947907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947887 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2eb25d-0a9e-4028-9c70-c8baec507384-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.947907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947892 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.948015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.947940 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f308b0-4938-435d-ac0d-9630aa87b92b-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.948015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.948010 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.948113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.948042 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.948187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.948164 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.948316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.948248 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmp4w\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-kube-api-access-fmp4w\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.948418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.948395 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.948694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.948667 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.948976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.948953 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.949091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.949004 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.949209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.949133 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.949806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.949771 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f308b0-4938-435d-ac0d-9630aa87b92b-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.951208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.951105 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.951399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.951374 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.951573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.951555 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.952009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.951984 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.970769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.970740 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vslj\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-kube-api-access-2vslj\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.971057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.971036 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:35.971854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.971834 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmp4w\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-kube-api-access-fmp4w\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:35.972235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:35.972217 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:36.015066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.012041 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:36.025634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.020697 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:36.378967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.378943 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk"] Oct 29 23:02:36.381964 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:02:36.381938 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2eb25d_0a9e_4028_9c70_c8baec507384.slice/crio-548d5e29561557efa4febf5a1fdf0035b6d02b73f53577350c8356089bd4fc83 WatchSource:0}: Error finding container 548d5e29561557efa4febf5a1fdf0035b6d02b73f53577350c8356089bd4fc83: Status 404 returned error can't find the container with id 548d5e29561557efa4febf5a1fdf0035b6d02b73f53577350c8356089bd4fc83 Oct 29 23:02:36.383985 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:02:36.383954 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f308b0_4938_435d_ac0d_9630aa87b92b.slice/crio-a52d489f0f7951d2d67b449c7795ce1c375e281ffb33fe74dffd8d0a820404d1 WatchSource:0}: Error finding container a52d489f0f7951d2d67b449c7795ce1c375e281ffb33fe74dffd8d0a820404d1: Status 404 returned error can't find the container with id a52d489f0f7951d2d67b449c7795ce1c375e281ffb33fe74dffd8d0a820404d1 Oct 29 23:02:36.384269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.384249 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5"] Oct 29 23:02:36.385448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.385417 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:36.385534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.385476 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:36.385534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.385511 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:36.387093 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.387067 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:36.387170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.387121 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:36.387170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.387148 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:36.766810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.766770 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" event={"ID":"f8f308b0-4938-435d-ac0d-9630aa87b92b","Type":"ContainerStarted","Data":"b0b46be0e0490d1ff8beac818bd38e02e72bb3d56dc3c5a47294b941b0a7efae"} Oct 29 23:02:36.767306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.766840 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" event={"ID":"f8f308b0-4938-435d-ac0d-9630aa87b92b","Type":"ContainerStarted","Data":"a52d489f0f7951d2d67b449c7795ce1c375e281ffb33fe74dffd8d0a820404d1"} Oct 29 23:02:36.769147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.769102 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" event={"ID":"8a2eb25d-0a9e-4028-9c70-c8baec507384","Type":"ContainerStarted","Data":"fd85efcbc7902309c023f716a2fad29f17316a8504046fd5bb706695e25a2c23"} Oct 29 23:02:36.769317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.769160 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" event={"ID":"8a2eb25d-0a9e-4028-9c70-c8baec507384","Type":"ContainerStarted","Data":"548d5e29561557efa4febf5a1fdf0035b6d02b73f53577350c8356089bd4fc83"} Oct 29 23:02:36.816306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.816228 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podStartSLOduration=1.816207662 podStartE2EDuration="1.816207662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:36.810789102 +0000 UTC m=+1153.926708719" watchObservedRunningTime="2026-10-29 23:02:36.816207662 +0000 UTC m=+1153.932127269" Oct 29 23:02:36.858157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:36.858097 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podStartSLOduration=1.858078392 podStartE2EDuration="1.858078392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:36.85482356 +0000 UTC m=+1153.970743165" watchObservedRunningTime="2026-10-29 23:02:36.858078392 +0000 UTC m=+1153.973997997" Oct 29 23:02:37.013249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:37.013217 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:37.015147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:37.015122 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:37.015242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:37.015167 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 23:02:37.021363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:37.021310 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:37.022691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:37.022665 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:02:37.022858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:37.022709 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 29 23:02:38.013245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:38.013211 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:38.013648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:38.013258 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 23:02:38.021620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:38.021596 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:02:38.021722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:38.021641 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 29 23:02:39.013496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:39.013463 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:39.013949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:39.013516 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 23:02:39.021392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:39.021362 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:02:39.021533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:39.021409 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 29 23:02:39.066605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:39.058641 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm"] Oct 29 23:02:39.082802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:39.082770 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6f8123-5t6mm"] Oct 29 23:02:40.012963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:40.012924 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:40.013146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:40.012984 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 23:02:40.021666 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:40.021640 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:02:40.022041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:40.021685 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 29 23:02:41.017621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:41.017220 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:41.018820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:41.018090 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 23:02:41.022197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:41.022158 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:02:41.027172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:41.027139 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 29 23:02:41.111405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:41.111321 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc262d7b-dbd9-4d57-b494-eeafa1cd450b" path="/var/lib/kubelet/pods/bc262d7b-dbd9-4d57-b494-eeafa1cd450b/volumes" Oct 29 23:02:41.191138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:41.191097 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84"] Oct 29 23:02:41.225541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:41.225507 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp"] Oct 29 23:02:42.015601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:42.012961 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:42.015601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:42.013021 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 23:02:42.032083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:42.032046 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:02:42.032649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:42.032618 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 29 23:02:42.808856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:42.808731 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7"] Oct 29 23:02:43.013492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:43.013062 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 29 23:02:43.013492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:43.013126 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 29 23:02:43.021754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:43.021722 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:02:43.021990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:43.021967 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 29 23:02:44.018439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:44.018413 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:44.018942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:44.018788 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:44.019471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:44.019452 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:02:44.026687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:44.026663 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:44.026986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:44.026969 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:44.027627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:44.027564 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:02:46.192009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.191970 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" podUID="c0c3f3c5-b591-4414-9be0-450aff82223d" containerName="payload-processing" containerID="cri-o://8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb" gracePeriod=25 Oct 29 23:02:46.226419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.226390 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" podUID="27742035-f9a1-4bf3-aacf-27483e0c4cad" containerName="payload-pre-processing" containerID="cri-o://824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635" gracePeriod=25 Oct 29 23:02:46.534190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.534154 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:02:46.574188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.574142 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:02:46.596649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.595633 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c3f3c5-b591-4414-9be0-450aff82223d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c3f3c5-b591-4414-9be0-450aff82223d-plugins-config-volume\") pod \"c0c3f3c5-b591-4414-9be0-450aff82223d\" (UID: \"c0c3f3c5-b591-4414-9be0-450aff82223d\") " Oct 29 23:02:46.596649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.595941 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c3f3c5-b591-4414-9be0-450aff82223d-kube-api-access-9fcxb\" (UniqueName: \"kubernetes.io/projected/c0c3f3c5-b591-4414-9be0-450aff82223d-kube-api-access-9fcxb\") pod \"c0c3f3c5-b591-4414-9be0-450aff82223d\" (UID: \"c0c3f3c5-b591-4414-9be0-450aff82223d\") " Oct 29 23:02:46.596649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.596011 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c3f3c5-b591-4414-9be0-450aff82223d-plugins-config-volume" pod "c0c3f3c5-b591-4414-9be0-450aff82223d" (UID: "c0c3f3c5-b591-4414-9be0-450aff82223d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:46.596882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.596653 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c3f3c5-b591-4414-9be0-450aff82223d-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:46.598801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.598773 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c3f3c5-b591-4414-9be0-450aff82223d-kube-api-access-9fcxb" pod "c0c3f3c5-b591-4414-9be0-450aff82223d" (UID: "c0c3f3c5-b591-4414-9be0-450aff82223d"). InnerVolumeSpecName "kube-api-access-9fcxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:46.698240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.698205 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27742035-f9a1-4bf3-aacf-27483e0c4cad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27742035-f9a1-4bf3-aacf-27483e0c4cad-plugins-config-volume\") pod \"27742035-f9a1-4bf3-aacf-27483e0c4cad\" (UID: \"27742035-f9a1-4bf3-aacf-27483e0c4cad\") " Oct 29 23:02:46.698410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.698395 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27742035-f9a1-4bf3-aacf-27483e0c4cad-kube-api-access-6cg9v\" (UniqueName: \"kubernetes.io/projected/27742035-f9a1-4bf3-aacf-27483e0c4cad-kube-api-access-6cg9v\") pod \"27742035-f9a1-4bf3-aacf-27483e0c4cad\" (UID: \"27742035-f9a1-4bf3-aacf-27483e0c4cad\") " Oct 29 23:02:46.698680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.698652 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27742035-f9a1-4bf3-aacf-27483e0c4cad-plugins-config-volume" pod "27742035-f9a1-4bf3-aacf-27483e0c4cad" (UID: "27742035-f9a1-4bf3-aacf-27483e0c4cad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:46.699194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.699176 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fcxb\" (UniqueName: \"kubernetes.io/projected/c0c3f3c5-b591-4414-9be0-450aff82223d-kube-api-access-9fcxb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:46.699239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.699197 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27742035-f9a1-4bf3-aacf-27483e0c4cad-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:46.700493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.700470 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27742035-f9a1-4bf3-aacf-27483e0c4cad-kube-api-access-6cg9v" pod "27742035-f9a1-4bf3-aacf-27483e0c4cad" (UID: "27742035-f9a1-4bf3-aacf-27483e0c4cad"). InnerVolumeSpecName "kube-api-access-6cg9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:46.804003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.803155 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cg9v\" (UniqueName: \"kubernetes.io/projected/27742035-f9a1-4bf3-aacf-27483e0c4cad-kube-api-access-6cg9v\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:46.977541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.977504 5736 generic.go:356] "Generic (PLEG): container finished" podID="c0c3f3c5-b591-4414-9be0-450aff82223d" containerID="8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb" exitCode=0 Oct 29 23:02:46.977749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.977547 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" event={"ID":"c0c3f3c5-b591-4414-9be0-450aff82223d","Type":"ContainerDied","Data":"8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb"} Oct 29 23:02:46.977749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.977557 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" Oct 29 23:02:46.977749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.977626 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84" event={"ID":"c0c3f3c5-b591-4414-9be0-450aff82223d","Type":"ContainerDied","Data":"2d577b188d1ee878ef4e8406dbe8c652b5bdc1223f5c2d4a2816a0bd14f80b2b"} Oct 29 23:02:46.977749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.977653 5736 scope.go:122] "RemoveContainer" containerID="8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb" Oct 29 23:02:46.980870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.980845 5736 generic.go:356] "Generic (PLEG): container finished" podID="27742035-f9a1-4bf3-aacf-27483e0c4cad" containerID="824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635" exitCode=0 Oct 29 23:02:46.981030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.980942 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" event={"ID":"27742035-f9a1-4bf3-aacf-27483e0c4cad","Type":"ContainerDied","Data":"824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635"} Oct 29 23:02:46.981030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.980980 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" event={"ID":"27742035-f9a1-4bf3-aacf-27483e0c4cad","Type":"ContainerDied","Data":"491897e82b69b89f2d0b3f8d486c2065d50fb3716df1f16e62371e407891bc8f"} Oct 29 23:02:46.981128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.981042 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp" Oct 29 23:02:46.996323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.996254 5736 scope.go:122] "RemoveContainer" containerID="8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb" Oct 29 23:02:46.996768 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:46.996738 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb\": container with ID starting with 8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb not found: ID does not exist" containerID="8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb" Oct 29 23:02:46.996886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.996779 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb"} err="failed to get container status \"8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb\": rpc error: code = NotFound desc = could not find container \"8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb\": container with ID starting with 8c1b4491ea5e1fbd98154a180fea7763425e61b86736e9af2b2d816cecb303cb not found: ID does not exist" Oct 29 23:02:46.996886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:46.996802 5736 scope.go:122] "RemoveContainer" containerID="824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635" Oct 29 23:02:47.012574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.012545 5736 scope.go:122] "RemoveContainer" containerID="824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635" Oct 29 23:02:47.012932 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:47.012886 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635\": container with ID starting with 824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635 not found: ID does not exist" containerID="824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635" Oct 29 23:02:47.013051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.012931 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635"} err="failed to get container status \"824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635\": rpc error: code = NotFound desc = could not find container \"824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635\": container with ID starting with 824d8b3ee9d50f06727eb8a88ca2c734ab252c478649a682f25702016d048635 not found: ID does not exist" Oct 29 23:02:47.035029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.034993 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp"] Oct 29 23:02:47.039462 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.039431 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7fa011-77fddb9cdc-fh6wp"] Oct 29 23:02:47.072277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.072193 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84"] Oct 29 23:02:47.079806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.079770 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7fa011-64677d6dfb-wnk84"] Oct 29 23:02:47.809723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.809676 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" podUID="954a1307-971f-41df-8c6b-1dc6c69b9495" containerName="payload-processing" containerID="cri-o://6fa252babc34f71dbd08df6dc2d4dcfdb38c2a1d6203e3d67eb3ad5ff621be24" gracePeriod=25 Oct 29 23:02:47.851747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.851708 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2"] Oct 29 23:02:47.863379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.863348 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9"] Oct 29 23:02:47.994451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.994405 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" event={"ID":"954a1307-971f-41df-8c6b-1dc6c69b9495","Type":"ContainerDied","Data":"6fa252babc34f71dbd08df6dc2d4dcfdb38c2a1d6203e3d67eb3ad5ff621be24"} Oct 29 23:02:47.994678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:47.994314 5736 generic.go:356] "Generic (PLEG): container finished" podID="954a1307-971f-41df-8c6b-1dc6c69b9495" containerID="6fa252babc34f71dbd08df6dc2d4dcfdb38c2a1d6203e3d67eb3ad5ff621be24" exitCode=0 Oct 29 23:02:48.133764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:48.133737 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 23:02:48.230669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:48.230374 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954a1307-971f-41df-8c6b-1dc6c69b9495-kube-api-access-84bpc\" (UniqueName: \"kubernetes.io/projected/954a1307-971f-41df-8c6b-1dc6c69b9495-kube-api-access-84bpc\") pod \"954a1307-971f-41df-8c6b-1dc6c69b9495\" (UID: \"954a1307-971f-41df-8c6b-1dc6c69b9495\") " Oct 29 23:02:48.231489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:48.231441 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954a1307-971f-41df-8c6b-1dc6c69b9495-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954a1307-971f-41df-8c6b-1dc6c69b9495-plugins-config-volume\") pod \"954a1307-971f-41df-8c6b-1dc6c69b9495\" (UID: \"954a1307-971f-41df-8c6b-1dc6c69b9495\") " Oct 29 23:02:48.233080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:48.232935 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954a1307-971f-41df-8c6b-1dc6c69b9495-plugins-config-volume" pod "954a1307-971f-41df-8c6b-1dc6c69b9495" (UID: "954a1307-971f-41df-8c6b-1dc6c69b9495"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:48.239151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:48.239117 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954a1307-971f-41df-8c6b-1dc6c69b9495-kube-api-access-84bpc" pod "954a1307-971f-41df-8c6b-1dc6c69b9495" (UID: "954a1307-971f-41df-8c6b-1dc6c69b9495"). InnerVolumeSpecName "kube-api-access-84bpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:48.337324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:48.337285 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84bpc\" (UniqueName: \"kubernetes.io/projected/954a1307-971f-41df-8c6b-1dc6c69b9495-kube-api-access-84bpc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:48.337324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:48.337325 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954a1307-971f-41df-8c6b-1dc6c69b9495-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.079239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:49.079202 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" Oct 29 23:02:49.095494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:49.093662 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27742035-f9a1-4bf3-aacf-27483e0c4cad" path="/var/lib/kubelet/pods/27742035-f9a1-4bf3-aacf-27483e0c4cad/volumes" Oct 29 23:02:49.095494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:49.094625 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c3f3c5-b591-4414-9be0-450aff82223d" path="/var/lib/kubelet/pods/c0c3f3c5-b591-4414-9be0-450aff82223d/volumes" Oct 29 23:02:49.095494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:49.095202 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7" event={"ID":"954a1307-971f-41df-8c6b-1dc6c69b9495","Type":"ContainerDied","Data":"2f65a9d777c910393016c662c3e14e8d8d3020a56f3f1d11177a161512dd6983"} Oct 29 23:02:49.095494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:49.095232 5736 scope.go:122] "RemoveContainer" containerID="6fa252babc34f71dbd08df6dc2d4dcfdb38c2a1d6203e3d67eb3ad5ff621be24" Oct 29 23:02:49.140405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:49.140326 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7"] Oct 29 23:02:49.154023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:49.153982 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-df82ee-77c45f865d-4mps7"] Oct 29 23:02:51.016604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:51.011771 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954a1307-971f-41df-8c6b-1dc6c69b9495" path="/var/lib/kubelet/pods/954a1307-971f-41df-8c6b-1dc6c69b9495/volumes" Oct 29 23:02:52.852261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:52.852205 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" podUID="29536dc6-b3a1-4aae-97ba-632cb4f2496e" containerName="payload-processing" containerID="cri-o://bfff6590d9571e07be9f2384c560456ce47d941f65ec813970b945a74bbed3af" gracePeriod=25 Oct 29 23:02:52.863735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:52.863704 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" podUID="e41baf80-0657-4e19-afe4-f66fd0aa3805" containerName="payload-pre-processing" containerID="cri-o://beaef7b9c5bbe8c004937c81812638e50c3438a2e4f87fb68da02008cea8598b" gracePeriod=25 Oct 29 23:02:53.131035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.131000 5736 generic.go:356] "Generic (PLEG): container finished" podID="e41baf80-0657-4e19-afe4-f66fd0aa3805" containerID="beaef7b9c5bbe8c004937c81812638e50c3438a2e4f87fb68da02008cea8598b" exitCode=0 Oct 29 23:02:53.131219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.131196 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" event={"ID":"e41baf80-0657-4e19-afe4-f66fd0aa3805","Type":"ContainerDied","Data":"beaef7b9c5bbe8c004937c81812638e50c3438a2e4f87fb68da02008cea8598b"} Oct 29 23:02:53.134612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.134543 5736 generic.go:356] "Generic (PLEG): container finished" podID="29536dc6-b3a1-4aae-97ba-632cb4f2496e" containerID="bfff6590d9571e07be9f2384c560456ce47d941f65ec813970b945a74bbed3af" exitCode=0 Oct 29 23:02:53.134789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.134771 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" event={"ID":"29536dc6-b3a1-4aae-97ba-632cb4f2496e","Type":"ContainerDied","Data":"bfff6590d9571e07be9f2384c560456ce47d941f65ec813970b945a74bbed3af"} Oct 29 23:02:53.176159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.176128 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:02:53.277858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.277817 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:02:53.371368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.371281 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29536dc6-b3a1-4aae-97ba-632cb4f2496e-kube-api-access-d5h8v\" (UniqueName: \"kubernetes.io/projected/29536dc6-b3a1-4aae-97ba-632cb4f2496e-kube-api-access-d5h8v\") pod \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\" (UID: \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\") " Oct 29 23:02:53.371528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.371498 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29536dc6-b3a1-4aae-97ba-632cb4f2496e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29536dc6-b3a1-4aae-97ba-632cb4f2496e-plugins-config-volume\") pod \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\" (UID: \"29536dc6-b3a1-4aae-97ba-632cb4f2496e\") " Oct 29 23:02:53.373187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.373142 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29536dc6-b3a1-4aae-97ba-632cb4f2496e-plugins-config-volume" pod "29536dc6-b3a1-4aae-97ba-632cb4f2496e" (UID: "29536dc6-b3a1-4aae-97ba-632cb4f2496e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:53.375218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.375188 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29536dc6-b3a1-4aae-97ba-632cb4f2496e-kube-api-access-d5h8v" pod "29536dc6-b3a1-4aae-97ba-632cb4f2496e" (UID: "29536dc6-b3a1-4aae-97ba-632cb4f2496e"). InnerVolumeSpecName "kube-api-access-d5h8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:53.473165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.473133 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41baf80-0657-4e19-afe4-f66fd0aa3805-kube-api-access-wv262\" (UniqueName: \"kubernetes.io/projected/e41baf80-0657-4e19-afe4-f66fd0aa3805-kube-api-access-wv262\") pod \"e41baf80-0657-4e19-afe4-f66fd0aa3805\" (UID: \"e41baf80-0657-4e19-afe4-f66fd0aa3805\") " Oct 29 23:02:53.473336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.473323 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41baf80-0657-4e19-afe4-f66fd0aa3805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41baf80-0657-4e19-afe4-f66fd0aa3805-plugins-config-volume\") pod \"e41baf80-0657-4e19-afe4-f66fd0aa3805\" (UID: \"e41baf80-0657-4e19-afe4-f66fd0aa3805\") " Oct 29 23:02:53.473757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.473722 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41baf80-0657-4e19-afe4-f66fd0aa3805-plugins-config-volume" pod "e41baf80-0657-4e19-afe4-f66fd0aa3805" (UID: "e41baf80-0657-4e19-afe4-f66fd0aa3805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:53.474033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.474009 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29536dc6-b3a1-4aae-97ba-632cb4f2496e-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:53.474033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.474029 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41baf80-0657-4e19-afe4-f66fd0aa3805-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:53.474229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.474039 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5h8v\" (UniqueName: \"kubernetes.io/projected/29536dc6-b3a1-4aae-97ba-632cb4f2496e-kube-api-access-d5h8v\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:53.475303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.475284 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41baf80-0657-4e19-afe4-f66fd0aa3805-kube-api-access-wv262" pod "e41baf80-0657-4e19-afe4-f66fd0aa3805" (UID: "e41baf80-0657-4e19-afe4-f66fd0aa3805"). InnerVolumeSpecName "kube-api-access-wv262". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:53.575213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:53.575176 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv262\" (UniqueName: \"kubernetes.io/projected/e41baf80-0657-4e19-afe4-f66fd0aa3805-kube-api-access-wv262\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:02:54.144868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.144780 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" Oct 29 23:02:54.144868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.144809 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9" event={"ID":"e41baf80-0657-4e19-afe4-f66fd0aa3805","Type":"ContainerDied","Data":"e8f72bb8b9eb68f6056df2ff3841da1450f23dd7c1203fa7fef76871f70983b1"} Oct 29 23:02:54.144868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.144861 5736 scope.go:122] "RemoveContainer" containerID="beaef7b9c5bbe8c004937c81812638e50c3438a2e4f87fb68da02008cea8598b" Oct 29 23:02:54.146491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.146466 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" event={"ID":"29536dc6-b3a1-4aae-97ba-632cb4f2496e","Type":"ContainerDied","Data":"970eb286c39b35e937a3e440f7c136b793b49c6e9bf8aefccc20334a2b346f7c"} Oct 29 23:02:54.146607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.146529 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2" Oct 29 23:02:54.158025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.158004 5736 scope.go:122] "RemoveContainer" containerID="bfff6590d9571e07be9f2384c560456ce47d941f65ec813970b945a74bbed3af" Oct 29 23:02:54.198121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.198093 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9"] Oct 29 23:02:54.209930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.209908 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-50734e-686c878c-hqzl9"] Oct 29 23:02:54.235590 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.235560 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2"] Oct 29 23:02:54.243687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:54.243660 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-50734e-6845ddcb6-22tl2"] Oct 29 23:02:55.007038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:55.007008 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29536dc6-b3a1-4aae-97ba-632cb4f2496e" path="/var/lib/kubelet/pods/29536dc6-b3a1-4aae-97ba-632cb4f2496e/volumes" Oct 29 23:02:55.007463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:02:55.007449 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41baf80-0657-4e19-afe4-f66fd0aa3805" path="/var/lib/kubelet/pods/e41baf80-0657-4e19-afe4-f66fd0aa3805/volumes" Oct 29 23:02:56.825912 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:02:56.825879 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:03:03.036329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:03.036292 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps"] Oct 29 23:03:03.069309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:03.069276 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6"] Oct 29 23:03:08.037564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.037444 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" podUID="39a22f14-1c4d-413c-988a-775d5a72b7d8" containerName="payload-processing" containerID="cri-o://560fb519185bafdd25f8796805ea35015dd2dc4e7b4c12f0aa01da3d5cf66157" gracePeriod=25 Oct 29 23:03:08.069678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.069536 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" podUID="f90f2c51-167f-4efa-a496-c30f745f6e06" containerName="payload-pre-processing" containerID="cri-o://528542598fb764a201599ce90e7cd6835b424f43ce86b17b9a02072f23fa0c1e" gracePeriod=25 Oct 29 23:03:08.277209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.277167 5736 generic.go:356] "Generic (PLEG): container finished" podID="f90f2c51-167f-4efa-a496-c30f745f6e06" containerID="528542598fb764a201599ce90e7cd6835b424f43ce86b17b9a02072f23fa0c1e" exitCode=0 Oct 29 23:03:08.277362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.277197 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" event={"ID":"f90f2c51-167f-4efa-a496-c30f745f6e06","Type":"ContainerDied","Data":"528542598fb764a201599ce90e7cd6835b424f43ce86b17b9a02072f23fa0c1e"} Oct 29 23:03:08.279202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.279174 5736 generic.go:356] "Generic (PLEG): container finished" podID="39a22f14-1c4d-413c-988a-775d5a72b7d8" containerID="560fb519185bafdd25f8796805ea35015dd2dc4e7b4c12f0aa01da3d5cf66157" exitCode=0 Oct 29 23:03:08.279324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.279265 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" event={"ID":"39a22f14-1c4d-413c-988a-775d5a72b7d8","Type":"ContainerDied","Data":"560fb519185bafdd25f8796805ea35015dd2dc4e7b4c12f0aa01da3d5cf66157"} Oct 29 23:03:08.379115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.379088 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:03:08.394936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.394912 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a22f14-1c4d-413c-988a-775d5a72b7d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a22f14-1c4d-413c-988a-775d5a72b7d8-plugins-config-volume\") pod \"39a22f14-1c4d-413c-988a-775d5a72b7d8\" (UID: \"39a22f14-1c4d-413c-988a-775d5a72b7d8\") " Oct 29 23:03:08.395061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.395013 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a22f14-1c4d-413c-988a-775d5a72b7d8-kube-api-access-9r66j\" (UniqueName: \"kubernetes.io/projected/39a22f14-1c4d-413c-988a-775d5a72b7d8-kube-api-access-9r66j\") pod \"39a22f14-1c4d-413c-988a-775d5a72b7d8\" (UID: \"39a22f14-1c4d-413c-988a-775d5a72b7d8\") " Oct 29 23:03:08.395261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.395237 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a22f14-1c4d-413c-988a-775d5a72b7d8-plugins-config-volume" pod "39a22f14-1c4d-413c-988a-775d5a72b7d8" (UID: "39a22f14-1c4d-413c-988a-775d5a72b7d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:08.395853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.395829 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a22f14-1c4d-413c-988a-775d5a72b7d8-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:08.397186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.397063 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a22f14-1c4d-413c-988a-775d5a72b7d8-kube-api-access-9r66j" pod "39a22f14-1c4d-413c-988a-775d5a72b7d8" (UID: "39a22f14-1c4d-413c-988a-775d5a72b7d8"). InnerVolumeSpecName "kube-api-access-9r66j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:08.409480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.409461 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:03:08.497306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.497272 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90f2c51-167f-4efa-a496-c30f745f6e06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f2c51-167f-4efa-a496-c30f745f6e06-plugins-config-volume\") pod \"f90f2c51-167f-4efa-a496-c30f745f6e06\" (UID: \"f90f2c51-167f-4efa-a496-c30f745f6e06\") " Oct 29 23:03:08.497501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.497488 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90f2c51-167f-4efa-a496-c30f745f6e06-kube-api-access-kvmjt\" (UniqueName: \"kubernetes.io/projected/f90f2c51-167f-4efa-a496-c30f745f6e06-kube-api-access-kvmjt\") pod \"f90f2c51-167f-4efa-a496-c30f745f6e06\" (UID: \"f90f2c51-167f-4efa-a496-c30f745f6e06\") " Oct 29 23:03:08.497680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.497654 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90f2c51-167f-4efa-a496-c30f745f6e06-plugins-config-volume" pod "f90f2c51-167f-4efa-a496-c30f745f6e06" (UID: "f90f2c51-167f-4efa-a496-c30f745f6e06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:08.498169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.498150 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r66j\" (UniqueName: \"kubernetes.io/projected/39a22f14-1c4d-413c-988a-775d5a72b7d8-kube-api-access-9r66j\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:08.498227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.498174 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90f2c51-167f-4efa-a496-c30f745f6e06-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:08.499727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.499698 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90f2c51-167f-4efa-a496-c30f745f6e06-kube-api-access-kvmjt" pod "f90f2c51-167f-4efa-a496-c30f745f6e06" (UID: "f90f2c51-167f-4efa-a496-c30f745f6e06"). InnerVolumeSpecName "kube-api-access-kvmjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:08.599530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:08.599443 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvmjt\" (UniqueName: \"kubernetes.io/projected/f90f2c51-167f-4efa-a496-c30f745f6e06-kube-api-access-kvmjt\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:09.288880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.288808 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" Oct 29 23:03:09.288880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.288843 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps" event={"ID":"39a22f14-1c4d-413c-988a-775d5a72b7d8","Type":"ContainerDied","Data":"9233e58f8531464fe7e2f4569979ace57571028b5d546700925e8752089bf12d"} Oct 29 23:03:09.289348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.288902 5736 scope.go:122] "RemoveContainer" containerID="560fb519185bafdd25f8796805ea35015dd2dc4e7b4c12f0aa01da3d5cf66157" Oct 29 23:03:09.290677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.290648 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" event={"ID":"f90f2c51-167f-4efa-a496-c30f745f6e06","Type":"ContainerDied","Data":"95c773a924392a616091d6ed08b64b43b720719089b63d6f1a2ecdfc83a98bca"} Oct 29 23:03:09.290783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.290655 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6" Oct 29 23:03:09.300592 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.300556 5736 scope.go:122] "RemoveContainer" containerID="528542598fb764a201599ce90e7cd6835b424f43ce86b17b9a02072f23fa0c1e" Oct 29 23:03:09.358257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.358221 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6"] Oct 29 23:03:09.365908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.365873 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5857bc4675-6ncl6"] Oct 29 23:03:09.394955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.394923 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps"] Oct 29 23:03:09.400825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:09.400798 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-796787ff6d-2hsps"] Oct 29 23:03:11.005698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:11.005667 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a22f14-1c4d-413c-988a-775d5a72b7d8" path="/var/lib/kubelet/pods/39a22f14-1c4d-413c-988a-775d5a72b7d8/volumes" Oct 29 23:03:11.006163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:11.006148 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90f2c51-167f-4efa-a496-c30f745f6e06" path="/var/lib/kubelet/pods/f90f2c51-167f-4efa-a496-c30f745f6e06/volumes" Oct 29 23:03:20.824822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:20.824788 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs"] Oct 29 23:03:20.863472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:20.863437 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7fa011-mp4zs"] Oct 29 23:03:21.006113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:21.006080 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb843c5d-abb5-4886-8085-2308e71fb2d0" path="/var/lib/kubelet/pods/eb843c5d-abb5-4886-8085-2308e71fb2d0/volumes" Oct 29 23:03:23.273651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.273623 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:03:23.274148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.273739 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:03:23.528497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.528416 5736 scope.go:122] "RemoveContainer" containerID="7c50e32a394067028b1fb18426831dd664033bf07b87b594631eafd31a25fcff" Oct 29 23:03:23.542105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.542067 5736 scope.go:122] "RemoveContainer" containerID="35fa16cf3aa96e5641e34ed0006172c332052e307b8cd127d41abf5f03973106" Oct 29 23:03:23.553495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.553469 5736 scope.go:122] "RemoveContainer" containerID="8ac9d633f78932fbcce759dcb2bf5f2a0768a21398984bba2f87e745594d6521" Oct 29 23:03:23.565107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.565087 5736 scope.go:122] "RemoveContainer" containerID="6981ad784b9d21288901fb6d4844402eafb5cbe2cb052bcd2cd826f88c973e29" Oct 29 23:03:23.576323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.576301 5736 scope.go:122] "RemoveContainer" containerID="9e6664f49a12d352470781355edd6f700ae6f9a7ba71d1a0d703be62b7c03453" Oct 29 23:03:23.589055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.589029 5736 scope.go:122] "RemoveContainer" containerID="3a273d22ff8c6c68a91b67aca75584256633a1837813305b45a69c6b5aea0156" Oct 29 23:03:23.601452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.601429 5736 scope.go:122] "RemoveContainer" containerID="bba32dfd1c70cd5e17aaa3b91b7ed28823f860e662ebb7630c40fc4155511cf6" Oct 29 23:03:23.611875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.611851 5736 scope.go:122] "RemoveContainer" containerID="09a760bf0641d280f1434a34ce75bd98dc6b9f33c7f1959d8def243e44285ab9" Oct 29 23:03:23.625194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.625169 5736 scope.go:122] "RemoveContainer" containerID="7b5c63be3eb2fbf050a30f964fc2eceb55a6b180a5c881f4099b32f1560126b3" Oct 29 23:03:23.638078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.638056 5736 scope.go:122] "RemoveContainer" containerID="87ca97b3d649405b391ba8534040ecb6891ca7aa006b3fd8fb532d5cd93ef53a" Oct 29 23:03:23.652619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.652590 5736 scope.go:122] "RemoveContainer" containerID="5f09dc578592d0712fb622d5c63775dabd6dc0d2ba41ec9fe75c3ca3ad99798d" Oct 29 23:03:23.664978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:23.664956 5736 scope.go:122] "RemoveContainer" containerID="06f67f8e2f951e027a6be39e3aa0a1a5ccec098429ac8b87ee1a7e8059bec040" Oct 29 23:03:25.831505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:25.831357 5736 generic.go:356] "Generic (PLEG): container finished" podID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerID="5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266" exitCode=0 Oct 29 23:03:25.835505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:25.831412 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" event={"ID":"fd9be3fc-62e6-4663-abff-5e89dcbfcf46","Type":"ContainerDied","Data":"5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266"} Oct 29 23:03:25.835505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:25.832266 5736 scope.go:122] "RemoveContainer" containerID="5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266" Oct 29 23:03:25.835505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:25.834668 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:25.835505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:25.834789 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:25.835505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:25.834841 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:26.825445 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:03:26.825404 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:03:26.906041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:26.906003 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" event={"ID":"fd9be3fc-62e6-4663-abff-5e89dcbfcf46","Type":"ContainerStarted","Data":"ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145"} Oct 29 23:03:26.920184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:26.920136 5736 generic.go:356] "Generic (PLEG): container finished" podID="e8984147-87bd-4105-989b-e2beb829eef4" containerID="eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7" exitCode=0 Oct 29 23:03:26.936735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:26.924826 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" event={"ID":"e8984147-87bd-4105-989b-e2beb829eef4","Type":"ContainerDied","Data":"eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7"} Oct 29 23:03:26.938080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:26.938058 5736 scope.go:122] "RemoveContainer" containerID="eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7" Oct 29 23:03:26.955624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:26.954702 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:26.955624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:26.954812 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:26.955624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:26.954861 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:27.726537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:27.726499 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:03:27.727883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:27.727854 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:27.728038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:27.727908 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 29 23:03:27.861716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:27.861679 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:03:27.977933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:27.976691 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" event={"ID":"e8984147-87bd-4105-989b-e2beb829eef4","Type":"ContainerStarted","Data":"4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb"} Oct 29 23:03:28.170611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:28.160825 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5"] Oct 29 23:03:28.187672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:28.178352 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add02620-jjqs5"] Oct 29 23:03:28.424944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:28.424851 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2"] Oct 29 23:03:28.425140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:28.425102 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" containerID="cri-o://74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd" gracePeriod=30 Oct 29 23:03:28.876004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:28.875548 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:28.876004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:28.875691 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 29 23:03:29.001361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:28.982023 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:03:29.082231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.082194 5736 patch_prober.go:29] interesting pod/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 29 23:03:29.082532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.082494 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 29 23:03:29.135461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.128793 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7" path="/var/lib/kubelet/pods/39bd9e21-dcf5-4fbe-a37a-0e2cdb04b1e7/volumes" Oct 29 23:03:29.726244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.726216 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:29.726471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.726449 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 29 23:03:29.883606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.883068 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459"] Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884229 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c0c3f3c5-b591-4414-9be0-450aff82223d" containerName="payload-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884255 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c3f3c5-b591-4414-9be0-450aff82223d" containerName="payload-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884285 5736 cpu_manager.go:414] "containerMap: removing container" podUID="39a22f14-1c4d-413c-988a-775d5a72b7d8" containerName="payload-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884294 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a22f14-1c4d-413c-988a-775d5a72b7d8" containerName="payload-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884320 5736 cpu_manager.go:414] "containerMap: removing container" podUID="29536dc6-b3a1-4aae-97ba-632cb4f2496e" containerName="payload-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884329 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29536dc6-b3a1-4aae-97ba-632cb4f2496e" containerName="payload-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884345 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f90f2c51-167f-4efa-a496-c30f745f6e06" containerName="payload-pre-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884353 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90f2c51-167f-4efa-a496-c30f745f6e06" containerName="payload-pre-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884373 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e41baf80-0657-4e19-afe4-f66fd0aa3805" containerName="payload-pre-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884381 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41baf80-0657-4e19-afe4-f66fd0aa3805" containerName="payload-pre-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884421 5736 cpu_manager.go:414] "containerMap: removing container" podUID="27742035-f9a1-4bf3-aacf-27483e0c4cad" containerName="payload-pre-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884429 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27742035-f9a1-4bf3-aacf-27483e0c4cad" containerName="payload-pre-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884478 5736 cpu_manager.go:414] "containerMap: removing container" podUID="954a1307-971f-41df-8c6b-1dc6c69b9495" containerName="payload-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884488 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954a1307-971f-41df-8c6b-1dc6c69b9495" containerName="payload-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884655 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="29536dc6-b3a1-4aae-97ba-632cb4f2496e" containerName="payload-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884671 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90f2c51-167f-4efa-a496-c30f745f6e06" containerName="payload-pre-processing" Oct 29 23:03:29.884659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884679 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41baf80-0657-4e19-afe4-f66fd0aa3805" containerName="payload-pre-processing" Oct 29 23:03:29.885646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884690 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="27742035-f9a1-4bf3-aacf-27483e0c4cad" containerName="payload-pre-processing" Oct 29 23:03:29.885646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884700 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="954a1307-971f-41df-8c6b-1dc6c69b9495" containerName="payload-processing" Oct 29 23:03:29.885646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884707 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c3f3c5-b591-4414-9be0-450aff82223d" containerName="payload-processing" Oct 29 23:03:29.885646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.884715 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a22f14-1c4d-413c-988a-775d5a72b7d8" containerName="payload-processing" Oct 29 23:03:29.887917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.887891 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:29.893273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.892375 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459"] Oct 29 23:03:29.906351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.906293 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:03:29.918125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.918089 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:03:29.995532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:29.995444 5736 patch_prober.go:29] interesting pod/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 29 23:03:30.012769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.012725 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 29 23:03:30.014366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.014331 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.014803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.014777 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.014983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.014970 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ksh\" (UniqueName: \"kubernetes.io/projected/d3d046e6-3059-4682-b985-b14ab9211fb8-kube-api-access-66ksh\") pod \"maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.117384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.117350 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.117638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.117621 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ksh\" (UniqueName: \"kubernetes.io/projected/d3d046e6-3059-4682-b985-b14ab9211fb8-kube-api-access-66ksh\") pod \"maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.117798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.117779 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.119070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.119049 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.122613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.121478 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.222884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.217297 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ksh\" (UniqueName: \"kubernetes.io/projected/d3d046e6-3059-4682-b985-b14ab9211fb8-kube-api-access-66ksh\") pod \"maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.536606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.535872 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:30.726044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.725961 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:30.726652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.726618 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 29 23:03:30.974386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.972541 5736 patch_prober.go:29] interesting pod/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 29 23:03:30.974386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:30.972622 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 29 23:03:31.266903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:31.256637 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459"] Oct 29 23:03:31.329624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:31.329547 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" event={"ID":"d3d046e6-3059-4682-b985-b14ab9211fb8","Type":"ContainerStarted","Data":"b89eeda01e050b46a2cd442dedf322ade39073ce2a42c6d6db85fe871491c65a"} Oct 29 23:03:31.726162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:31.726133 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:31.726396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:31.726370 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 29 23:03:31.960597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:31.960150 5736 patch_prober.go:29] interesting pod/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 29 23:03:31.960597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:31.960211 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 29 23:03:32.380600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:32.379906 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" event={"ID":"d3d046e6-3059-4682-b985-b14ab9211fb8","Type":"ContainerStarted","Data":"3df52e5aaef8e7398038d04e8a160d51e352cf1d2be8f31b40e486a8124007a6"} Oct 29 23:03:32.435605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:32.434910 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" podStartSLOduration=3.434891048 podStartE2EDuration="3.434891048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:32.431994803 +0000 UTC m=+1209.547914398" watchObservedRunningTime="2026-10-29 23:03:32.434891048 +0000 UTC m=+1209.550810656" Oct 29 23:03:32.726070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:32.726034 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:32.726224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:32.726087 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 29 23:03:32.969611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:32.967756 5736 patch_prober.go:29] interesting pod/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 29 23:03:32.969611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:32.967941 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 29 23:03:33.404991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.404716 5736 generic.go:356] "Generic (PLEG): container finished" podID="d3d046e6-3059-4682-b985-b14ab9211fb8" containerID="3df52e5aaef8e7398038d04e8a160d51e352cf1d2be8f31b40e486a8124007a6" exitCode=0 Oct 29 23:03:33.404991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.404835 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" event={"ID":"d3d046e6-3059-4682-b985-b14ab9211fb8","Type":"ContainerDied","Data":"3df52e5aaef8e7398038d04e8a160d51e352cf1d2be8f31b40e486a8124007a6"} Oct 29 23:03:33.755682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.740880 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:03:33.756212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.756144 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:03:33.765525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.765506 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:03:33.800946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.800921 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-envoy\") pod \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " Oct 29 23:03:33.801049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.800957 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-credential-socket\") pod \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " Oct 29 23:03:33.801049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.800992 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-podinfo\") pod \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " Oct 29 23:03:33.801140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801048 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istiod-ca-cert\") pod \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " Oct 29 23:03:33.801272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801249 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-certs\") pod \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " Oct 29 23:03:33.801272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801266 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-credential-socket" pod "c57fe8cc-c1f6-4f0d-8a0e-f56430402465" (UID: "c57fe8cc-c1f6-4f0d-8a0e-f56430402465"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:33.801467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801286 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-kube-api-access-mxzfc\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-kube-api-access-mxzfc\") pod \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " Oct 29 23:03:33.801467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801369 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-data\") pod \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " Oct 29 23:03:33.801618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801404 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istiod-ca-cert" pod "c57fe8cc-c1f6-4f0d-8a0e-f56430402465" (UID: "c57fe8cc-c1f6-4f0d-8a0e-f56430402465"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:33.801618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801413 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-socket\") pod \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " Oct 29 23:03:33.801748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801490 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-certs" pod "c57fe8cc-c1f6-4f0d-8a0e-f56430402465" (UID: "c57fe8cc-c1f6-4f0d-8a0e-f56430402465"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:33.801748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801664 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-data" pod "c57fe8cc-c1f6-4f0d-8a0e-f56430402465" (UID: "c57fe8cc-c1f6-4f0d-8a0e-f56430402465"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:33.801748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801675 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-socket" pod "c57fe8cc-c1f6-4f0d-8a0e-f56430402465" (UID: "c57fe8cc-c1f6-4f0d-8a0e-f56430402465"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:33.801748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.801667 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-token\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-token\") pod \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\" (UID: \"c57fe8cc-c1f6-4f0d-8a0e-f56430402465\") " Oct 29 23:03:33.802457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.802432 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:33.802457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.802460 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:33.802667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.802473 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:33.802667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.802487 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:33.802667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.802499 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:33.803560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.803535 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-envoy" pod "c57fe8cc-c1f6-4f0d-8a0e-f56430402465" (UID: "c57fe8cc-c1f6-4f0d-8a0e-f56430402465"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:33.803680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.803647 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-kube-api-access-mxzfc" pod "c57fe8cc-c1f6-4f0d-8a0e-f56430402465" (UID: "c57fe8cc-c1f6-4f0d-8a0e-f56430402465"). InnerVolumeSpecName "kube-api-access-mxzfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:33.803815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.803797 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-podinfo" pod "c57fe8cc-c1f6-4f0d-8a0e-f56430402465" (UID: "c57fe8cc-c1f6-4f0d-8a0e-f56430402465"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:33.803858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.803829 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-token" pod "c57fe8cc-c1f6-4f0d-8a0e-f56430402465" (UID: "c57fe8cc-c1f6-4f0d-8a0e-f56430402465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:33.919194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.919160 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:33.919194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.919195 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:33.919437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.919213 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxzfc\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-kube-api-access-mxzfc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:33.919437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.919227 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57fe8cc-c1f6-4f0d-8a0e-f56430402465-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:33.972748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.972711 5736 patch_prober.go:29] interesting pod/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 29 23:03:33.973001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:33.972972 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 29 23:03:34.505055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.495952 5736 generic.go:356] "Generic (PLEG): container finished" podID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerID="74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd" exitCode=0 Oct 29 23:03:34.505055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.496327 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" event={"ID":"c57fe8cc-c1f6-4f0d-8a0e-f56430402465","Type":"ContainerDied","Data":"74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd"} Oct 29 23:03:34.505055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.496437 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" event={"ID":"c57fe8cc-c1f6-4f0d-8a0e-f56430402465","Type":"ContainerDied","Data":"cbc1e67dc1984dd7961eacbd2811a62df005d6a78114fc2150bc5b8215714c73"} Oct 29 23:03:34.505055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.496478 5736 scope.go:122] "RemoveContainer" containerID="74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd" Oct 29 23:03:34.505055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.497513 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2" Oct 29 23:03:34.667749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.664882 5736 scope.go:122] "RemoveContainer" containerID="9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b" Oct 29 23:03:34.772737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.770755 5736 scope.go:122] "RemoveContainer" containerID="74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd" Oct 29 23:03:34.772737 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:03:34.771233 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd\": container with ID starting with 74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd not found: ID does not exist" containerID="74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd" Oct 29 23:03:34.772737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.771274 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd"} err="failed to get container status \"74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd\": rpc error: code = NotFound desc = could not find container \"74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd\": container with ID starting with 74e5b7035865861f6b9b9ccd99810cc342bcf87f57cf6bf7f63bccfd039236cd not found: ID does not exist" Oct 29 23:03:34.772737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.771297 5736 scope.go:122] "RemoveContainer" containerID="9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b" Oct 29 23:03:34.772737 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:03:34.772316 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b\": container with ID starting with 9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b not found: ID does not exist" containerID="9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b" Oct 29 23:03:34.772737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.772350 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b"} err="failed to get container status \"9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b\": rpc error: code = NotFound desc = could not find container \"9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b\": container with ID starting with 9374d95d7df62a2daf5c7631bba6f8308a85a1fcf9d443992efdfcc1c011460b not found: ID does not exist" Oct 29 23:03:34.819359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.819270 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2"] Oct 29 23:03:34.828235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.828202 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7fa011-openshift-default-ccd5cdb6c-jrbg2"] Oct 29 23:03:34.994611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:34.993191 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:35.008880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.006953 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" path="/var/lib/kubelet/pods/c57fe8cc-c1f6-4f0d-8a0e-f56430402465/volumes" Oct 29 23:03:35.070772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.066707 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:03:35.070772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.069480 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:03:35.070772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.070722 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:03:35.101691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.101654 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-xks-ca\") pod \"d3d046e6-3059-4682-b985-b14ab9211fb8\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " Oct 29 23:03:35.101860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.101720 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-openshift-ca\") pod \"d3d046e6-3059-4682-b985-b14ab9211fb8\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " Oct 29 23:03:35.102004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.101984 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d046e6-3059-4682-b985-b14ab9211fb8-kube-api-access-66ksh\" (UniqueName: \"kubernetes.io/projected/d3d046e6-3059-4682-b985-b14ab9211fb8-kube-api-access-66ksh\") pod \"d3d046e6-3059-4682-b985-b14ab9211fb8\" (UID: \"d3d046e6-3059-4682-b985-b14ab9211fb8\") " Oct 29 23:03:35.102059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.102040 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-openshift-ca" pod "d3d046e6-3059-4682-b985-b14ab9211fb8" (UID: "d3d046e6-3059-4682-b985-b14ab9211fb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:35.103021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.102999 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:35.104146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.104123 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-xks-ca" pod "d3d046e6-3059-4682-b985-b14ab9211fb8" (UID: "d3d046e6-3059-4682-b985-b14ab9211fb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:35.104704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.104675 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d046e6-3059-4682-b985-b14ab9211fb8-kube-api-access-66ksh" pod "d3d046e6-3059-4682-b985-b14ab9211fb8" (UID: "d3d046e6-3059-4682-b985-b14ab9211fb8"). InnerVolumeSpecName "kube-api-access-66ksh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:35.205324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.205283 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d046e6-3059-4682-b985-b14ab9211fb8-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:35.205324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.205324 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66ksh\" (UniqueName: \"kubernetes.io/projected/d3d046e6-3059-4682-b985-b14ab9211fb8-kube-api-access-66ksh\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:35.510479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.510449 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" Oct 29 23:03:35.511014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.510490 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459" event={"ID":"d3d046e6-3059-4682-b985-b14ab9211fb8","Type":"ContainerDied","Data":"b89eeda01e050b46a2cd442dedf322ade39073ce2a42c6d6db85fe871491c65a"} Oct 29 23:03:35.511014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:35.510550 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89eeda01e050b46a2cd442dedf322ade39073ce2a42c6d6db85fe871491c65a" Oct 29 23:03:36.617815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.617777 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w"] Oct 29 23:03:36.618526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.618509 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" Oct 29 23:03:36.618526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.618526 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" Oct 29 23:03:36.618743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.618560 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d3d046e6-3059-4682-b985-b14ab9211fb8" containerName="revoke-keys" Oct 29 23:03:36.618743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.618566 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d046e6-3059-4682-b985-b14ab9211fb8" containerName="revoke-keys" Oct 29 23:03:36.618856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.618754 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" Oct 29 23:03:36.618856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.618775 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" Oct 29 23:03:36.618856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.618789 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d046e6-3059-4682-b985-b14ab9211fb8" containerName="revoke-keys" Oct 29 23:03:36.622711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.622687 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.632969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.632942 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.633089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.633070 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.633172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.633153 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.633300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.633279 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.633942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.633476 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.634108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.634086 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.634207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.634125 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.634207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.634149 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v649c\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-kube-api-access-v649c\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.637364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.637277 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm"] Oct 29 23:03:36.637761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.637732 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.638083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.638066 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" Oct 29 23:03:36.638151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.638085 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57fe8cc-c1f6-4f0d-8a0e-f56430402465" containerName="istio-proxy" Oct 29 23:03:36.640643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.640625 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.644468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.644447 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-10b990-openshift-default-dockercfg-nnpxx\"" Oct 29 23:03:36.650271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.650149 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w"] Oct 29 23:03:36.670632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.669826 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm"] Oct 29 23:03:36.739337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739304 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.739486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739353 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.739486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739385 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.739486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739419 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.739687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739608 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.739687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739655 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v649c\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-kube-api-access-v649c\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.739801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739689 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.739801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739719 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.739801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739760 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.739801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739790 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.739962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739798 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvl6s\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-kube-api-access-zvl6s\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.739962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739865 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.739962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739925 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.740076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.739990 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.740076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.740069 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.740185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.740098 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.740242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.740173 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.740242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.740214 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.740436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.740240 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.740436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.740302 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.740561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.740441 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c1dd77-c258-44c2-be49-ad19811704e9-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.740561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.740504 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.740561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.740541 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c1dd77-c258-44c2-be49-ad19811704e9-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.742111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.742091 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.742559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.742537 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.763099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.763063 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.763200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.763141 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v649c\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-kube-api-access-v649c\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.842661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.842626 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c1dd77-c258-44c2-be49-ad19811704e9-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.842803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.842749 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.842854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.842806 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.842906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.842883 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.842960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.842932 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvl6s\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-kube-api-access-zvl6s\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.843022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.842992 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.843202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.843089 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.843304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.843223 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.843304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.843270 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.843414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.843284 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.843414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.843363 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c1dd77-c258-44c2-be49-ad19811704e9-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.843730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.843701 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.844136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.844108 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c1dd77-c258-44c2-be49-ad19811704e9-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.844254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.844228 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.845770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.845744 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c1dd77-c258-44c2-be49-ad19811704e9-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.845881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.845857 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.866692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.866668 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvl6s\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-kube-api-access-zvl6s\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.866798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.866775 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:36.938902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.938844 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:36.958268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:36.958244 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:37.283765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.283724 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm"] Oct 29 23:03:37.286123 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:03:37.286092 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c1dd77_c258_44c2_be49_ad19811704e9.slice/crio-fb53ff32beee70fd87af65bf343aef08f0057e31909966fdfc81e49f3521f092 WatchSource:0}: Error finding container fb53ff32beee70fd87af65bf343aef08f0057e31909966fdfc81e49f3521f092: Status 404 returned error can't find the container with id fb53ff32beee70fd87af65bf343aef08f0057e31909966fdfc81e49f3521f092 Oct 29 23:03:37.288057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.288033 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w"] Oct 29 23:03:37.288867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.288835 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:37.288945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.288902 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:37.288945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.288931 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:37.301559 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:03:37.301536 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcec88bf_6f9f_4d99_bd0b_0ec5c6dd9a81.slice/crio-0f53ca92487abfcaedac69470e91e53acd30e2dfa89077e2a462e0b9e9b665a7 WatchSource:0}: Error finding container 0f53ca92487abfcaedac69470e91e53acd30e2dfa89077e2a462e0b9e9b665a7: Status 404 returned error can't find the container with id 0f53ca92487abfcaedac69470e91e53acd30e2dfa89077e2a462e0b9e9b665a7 Oct 29 23:03:37.304109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.304079 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:37.304183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.304145 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:37.304226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.304191 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:03:37.531262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.531216 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" event={"ID":"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81","Type":"ContainerStarted","Data":"adddc641767b19c24160b0f64d2b812c8e8673a077b3f20091e2cb813e345378"} Oct 29 23:03:37.531453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.531288 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" event={"ID":"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81","Type":"ContainerStarted","Data":"0f53ca92487abfcaedac69470e91e53acd30e2dfa89077e2a462e0b9e9b665a7"} Oct 29 23:03:37.533868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.533791 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" event={"ID":"43c1dd77-c258-44c2-be49-ad19811704e9","Type":"ContainerStarted","Data":"12369565caf30742ac467486e0177197ffbc1a118a9c230731fd62748e990e18"} Oct 29 23:03:37.533868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.533851 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" event={"ID":"43c1dd77-c258-44c2-be49-ad19811704e9","Type":"ContainerStarted","Data":"fb53ff32beee70fd87af65bf343aef08f0057e31909966fdfc81e49f3521f092"} Oct 29 23:03:37.582232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.582176 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" podStartSLOduration=1.5821574539999999 podStartE2EDuration="1.582157454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:37.576259772 +0000 UTC m=+1214.692179384" watchObservedRunningTime="2026-10-29 23:03:37.582157454 +0000 UTC m=+1214.698077064" Oct 29 23:03:37.636065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.636002 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" podStartSLOduration=1.635982322 podStartE2EDuration="1.635982322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:37.632827662 +0000 UTC m=+1214.748747269" watchObservedRunningTime="2026-10-29 23:03:37.635982322 +0000 UTC m=+1214.751901929" Oct 29 23:03:37.940783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.940692 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:37.959599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:37.958597 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:38.984316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:38.984280 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:38.984911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:38.984879 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 23:03:38.985185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:38.984619 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:03:38.985271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:38.985245 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 23:03:39.811318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:39.811149 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9"] Oct 29 23:03:39.812085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:39.812048 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" podUID="7974b8ff-aa29-452f-b67e-1729a1ac9f91" containerName="maas-api" containerID="cri-o://30e05fe19838356d9717dc07c990cd5c51f8af885770eacd014dfbffb83b3ec0" gracePeriod=30 Oct 29 23:03:39.985607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:39.982876 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:03:39.985607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:39.982937 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 23:03:39.985607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:39.983332 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:39.985607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:39.983365 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 23:03:40.787660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:40.783801 5736 generic.go:356] "Generic (PLEG): container finished" podID="7974b8ff-aa29-452f-b67e-1729a1ac9f91" containerID="30e05fe19838356d9717dc07c990cd5c51f8af885770eacd014dfbffb83b3ec0" exitCode=0 Oct 29 23:03:40.787660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:40.784122 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" event={"ID":"7974b8ff-aa29-452f-b67e-1729a1ac9f91","Type":"ContainerDied","Data":"30e05fe19838356d9717dc07c990cd5c51f8af885770eacd014dfbffb83b3ec0"} Oct 29 23:03:40.840599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:40.835327 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:40.952608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:40.952417 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:40.952608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:40.952480 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 23:03:40.968042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:40.968009 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:03:40.968292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:40.968268 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 23:03:40.991282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:40.990825 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:41.192661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.185656 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:41.223522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.223487 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8"] Oct 29 23:03:41.223716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.223541 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df82ee-4wwb8"] Oct 29 23:03:41.831649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.829789 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:03:41.852611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.851505 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" event={"ID":"7974b8ff-aa29-452f-b67e-1729a1ac9f91","Type":"ContainerDied","Data":"ad9592543da305538033c8ba59f263cebac5fcada713bd449a3bb5b9105edc0d"} Oct 29 23:03:41.852611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.851592 5736 scope.go:122] "RemoveContainer" containerID="30e05fe19838356d9717dc07c990cd5c51f8af885770eacd014dfbffb83b3ec0" Oct 29 23:03:41.852611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.851765 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9" Oct 29 23:03:41.921666 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.913236 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7974b8ff-aa29-452f-b67e-1729a1ac9f91-kube-api-access-s5jtd\" (UniqueName: \"kubernetes.io/projected/7974b8ff-aa29-452f-b67e-1729a1ac9f91-kube-api-access-s5jtd\") pod \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " Oct 29 23:03:41.922134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.922107 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7974b8ff-aa29-452f-b67e-1729a1ac9f91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7974b8ff-aa29-452f-b67e-1729a1ac9f91-openshift-service-ca\") pod \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " Oct 29 23:03:41.922349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.922328 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-maas-api-tls\") pod \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " Oct 29 23:03:41.922525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.922498 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-metrics-certs\") pod \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\" (UID: \"7974b8ff-aa29-452f-b67e-1729a1ac9f91\") " Oct 29 23:03:41.925946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.925920 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7974b8ff-aa29-452f-b67e-1729a1ac9f91-openshift-service-ca" pod "7974b8ff-aa29-452f-b67e-1729a1ac9f91" (UID: "7974b8ff-aa29-452f-b67e-1729a1ac9f91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:41.982606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:41.959181 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7974b8ff-aa29-452f-b67e-1729a1ac9f91-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:42.047679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.031133 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7974b8ff-aa29-452f-b67e-1729a1ac9f91-kube-api-access-s5jtd" pod "7974b8ff-aa29-452f-b67e-1729a1ac9f91" (UID: "7974b8ff-aa29-452f-b67e-1729a1ac9f91"). InnerVolumeSpecName "kube-api-access-s5jtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:42.047679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.031703 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-maas-api-tls" pod "7974b8ff-aa29-452f-b67e-1729a1ac9f91" (UID: "7974b8ff-aa29-452f-b67e-1729a1ac9f91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:42.065856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.065297 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:42.065856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.065337 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jtd\" (UniqueName: \"kubernetes.io/projected/7974b8ff-aa29-452f-b67e-1729a1ac9f91-kube-api-access-s5jtd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:42.126332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.126290 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-metrics-certs" pod "7974b8ff-aa29-452f-b67e-1729a1ac9f91" (UID: "7974b8ff-aa29-452f-b67e-1729a1ac9f91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:42.208563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.129142 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:42.208563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.207929 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 23:03:42.208563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.129183 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:03:42.208563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.208410 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 23:03:42.324505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.324468 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7974b8ff-aa29-452f-b67e-1729a1ac9f91-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:42.385605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.355669 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:42.385605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.356067 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 23:03:42.385605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.371310 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:42.453604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.442539 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp"] Oct 29 23:03:42.453604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.445804 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7974b8ff-aa29-452f-b67e-1729a1ac9f91" containerName="maas-api" Oct 29 23:03:42.453604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.445832 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7974b8ff-aa29-452f-b67e-1729a1ac9f91" containerName="maas-api" Oct 29 23:03:42.494940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.494886 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7974b8ff-aa29-452f-b67e-1729a1ac9f91" containerName="maas-api" Oct 29 23:03:42.524960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.524907 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:42.591767 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:03:42.583697 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7974b8ff_aa29_452f_b67e_1729a1ac9f91.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7974b8ff_aa29_452f_b67e_1729a1ac9f91.slice/crio-ad9592543da305538033c8ba59f263cebac5fcada713bd449a3bb5b9105edc0d\": RecentStats: unable to find data in memory cache]" Oct 29 23:03:42.592648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.592164 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.592777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.592074 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp"] Oct 29 23:03:42.608343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.608311 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:03:42.608467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.608359 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:03:42.608709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.608689 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9"] Oct 29 23:03:42.614958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.614936 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-df82ee-bf4fcc9f7-mwhw9"] Oct 29 23:03:42.685622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.682931 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkvwg\" (UniqueName: \"kubernetes.io/projected/543f43db-04e2-4d24-a8cc-97815e21a22c-kube-api-access-zkvwg\") pod \"maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.685622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.683898 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.685622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.684059 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.785996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.785899 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.785996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.785960 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.786254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.786079 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkvwg\" (UniqueName: \"kubernetes.io/projected/543f43db-04e2-4d24-a8cc-97815e21a22c-kube-api-access-zkvwg\") pod \"maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.786768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.786739 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.789330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.789303 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.809832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.809808 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkvwg\" (UniqueName: \"kubernetes.io/projected/543f43db-04e2-4d24-a8cc-97815e21a22c-kube-api-access-zkvwg\") pod \"maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.915616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.910558 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:42.986606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.984513 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:03:42.986606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.984570 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 23:03:42.986606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.984978 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:42.986606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:42.985017 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 23:03:43.020035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:43.020002 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099592c0-0b2d-4ea6-a239-2d98a92e52c0" path="/var/lib/kubelet/pods/099592c0-0b2d-4ea6-a239-2d98a92e52c0/volumes" Oct 29 23:03:43.020501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:43.020486 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7974b8ff-aa29-452f-b67e-1729a1ac9f91" path="/var/lib/kubelet/pods/7974b8ff-aa29-452f-b67e-1729a1ac9f91/volumes" Oct 29 23:03:43.995604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:43.992209 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:43.995604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:43.992262 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:03:43.995604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:43.992269 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 23:03:43.995604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:43.992305 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 23:03:44.052832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.052795 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp"] Oct 29 23:03:44.053324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.053295 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" event={"ID":"543f43db-04e2-4d24-a8cc-97815e21a22c","Type":"ContainerStarted","Data":"f9eea95273a14ea039d883ffb917b3d007e272b5869f9c8b31799ed3c7339e99"} Oct 29 23:03:44.373654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.362724 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:44.373654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.367661 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:44.381763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.381720 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 29 23:03:44.415415 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.411350 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:44.598293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.585650 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:44.678788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.665430 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:44.719509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.719464 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:44.939747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.939715 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:03:44.939897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.939771 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 23:03:44.959511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.959476 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 29 23:03:44.959676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:44.959536 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 29 23:03:45.048619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:45.048571 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" event={"ID":"543f43db-04e2-4d24-a8cc-97815e21a22c","Type":"ContainerStarted","Data":"8aaf0818af9ce92cf0f163ec3da1aae1cee724a46979d345c05f18053812e77d"} Oct 29 23:03:45.944737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:45.944658 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:45.945139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:45.945104 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:45.945909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:45.945886 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:03:45.963017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:45.962996 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:45.963241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:45.963225 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:45.963892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:45.963874 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:03:46.016118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:46.016074 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" podStartSLOduration=4.01605981 podStartE2EDuration="4.01605981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:45.091651491 +0000 UTC m=+1222.207571097" watchObservedRunningTime="2026-10-29 23:03:46.01605981 +0000 UTC m=+1223.131979434" Oct 29 23:03:46.060823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:46.060789 5736 generic.go:356] "Generic (PLEG): container finished" podID="543f43db-04e2-4d24-a8cc-97815e21a22c" containerID="8aaf0818af9ce92cf0f163ec3da1aae1cee724a46979d345c05f18053812e77d" exitCode=0 Oct 29 23:03:46.061355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:46.060818 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" event={"ID":"543f43db-04e2-4d24-a8cc-97815e21a22c","Type":"ContainerDied","Data":"8aaf0818af9ce92cf0f163ec3da1aae1cee724a46979d345c05f18053812e77d"} Oct 29 23:03:47.038884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.038853 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl"] Oct 29 23:03:47.050027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.049996 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ed7124-q89wl"] Oct 29 23:03:47.273650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.273621 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:47.386507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.380301 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-openshift-ca\") pod \"543f43db-04e2-4d24-a8cc-97815e21a22c\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " Oct 29 23:03:47.386507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.380451 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543f43db-04e2-4d24-a8cc-97815e21a22c-kube-api-access-zkvwg\" (UniqueName: \"kubernetes.io/projected/543f43db-04e2-4d24-a8cc-97815e21a22c-kube-api-access-zkvwg\") pod \"543f43db-04e2-4d24-a8cc-97815e21a22c\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " Oct 29 23:03:47.386507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.380507 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-xks-ca\") pod \"543f43db-04e2-4d24-a8cc-97815e21a22c\" (UID: \"543f43db-04e2-4d24-a8cc-97815e21a22c\") " Oct 29 23:03:47.386507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.382641 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-openshift-ca" pod "543f43db-04e2-4d24-a8cc-97815e21a22c" (UID: "543f43db-04e2-4d24-a8cc-97815e21a22c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:47.399645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.392553 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543f43db-04e2-4d24-a8cc-97815e21a22c-kube-api-access-zkvwg" pod "543f43db-04e2-4d24-a8cc-97815e21a22c" (UID: "543f43db-04e2-4d24-a8cc-97815e21a22c"). InnerVolumeSpecName "kube-api-access-zkvwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:47.399645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.398999 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-xks-ca" pod "543f43db-04e2-4d24-a8cc-97815e21a22c" (UID: "543f43db-04e2-4d24-a8cc-97815e21a22c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:47.483018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.482979 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkvwg\" (UniqueName: \"kubernetes.io/projected/543f43db-04e2-4d24-a8cc-97815e21a22c-kube-api-access-zkvwg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.483018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.483021 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.483220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:47.483036 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543f43db-04e2-4d24-a8cc-97815e21a22c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:48.079064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:48.079037 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" Oct 29 23:03:48.079322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:48.079033 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp" event={"ID":"543f43db-04e2-4d24-a8cc-97815e21a22c","Type":"ContainerDied","Data":"f9eea95273a14ea039d883ffb917b3d007e272b5869f9c8b31799ed3c7339e99"} Oct 29 23:03:48.079435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:48.079346 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9eea95273a14ea039d883ffb917b3d007e272b5869f9c8b31799ed3c7339e99" Oct 29 23:03:49.007125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.007088 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8363fe43-b47f-42d2-9ce2-d7c73ec010a6" path="/var/lib/kubelet/pods/8363fe43-b47f-42d2-9ce2-d7c73ec010a6/volumes" Oct 29 23:03:49.306683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.306604 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr"] Oct 29 23:03:49.307304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.307289 5736 cpu_manager.go:414] "containerMap: removing container" podUID="543f43db-04e2-4d24-a8cc-97815e21a22c" containerName="revoke-keys" Oct 29 23:03:49.307353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.307305 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543f43db-04e2-4d24-a8cc-97815e21a22c" containerName="revoke-keys" Oct 29 23:03:49.307452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.307443 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="543f43db-04e2-4d24-a8cc-97815e21a22c" containerName="revoke-keys" Oct 29 23:03:49.310467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.310450 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.326544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.326523 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:03:49.340531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.340497 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:03:49.346358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.346334 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr"] Oct 29 23:03:49.430336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.430308 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.430524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.430502 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.430714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.430698 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cpc\" (UniqueName: \"kubernetes.io/projected/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-kube-api-access-58cpc\") pod \"maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.532152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.532114 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.532315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.532293 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cpc\" (UniqueName: \"kubernetes.io/projected/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-kube-api-access-58cpc\") pod \"maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.532396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.532382 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.533032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.533009 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.534449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.534430 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.556153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.556125 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cpc\" (UniqueName: \"kubernetes.io/projected/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-kube-api-access-58cpc\") pod \"maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.624127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.624074 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:49.967633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:49.967608 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr"] Oct 29 23:03:49.969859 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:03:49.969833 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877b7cf3_5ac1_4e97_b2bc_b0a3129e2016.slice/crio-c90ce4c9e20c7a223245d2ffb39be3f2aae69a16ca4d39a3d573066f1320c3eb WatchSource:0}: Error finding container c90ce4c9e20c7a223245d2ffb39be3f2aae69a16ca4d39a3d573066f1320c3eb: Status 404 returned error can't find the container with id c90ce4c9e20c7a223245d2ffb39be3f2aae69a16ca4d39a3d573066f1320c3eb Oct 29 23:03:50.098398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:50.098365 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" event={"ID":"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016","Type":"ContainerStarted","Data":"c90ce4c9e20c7a223245d2ffb39be3f2aae69a16ca4d39a3d573066f1320c3eb"} Oct 29 23:03:51.163244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:51.162710 5736 generic.go:356] "Generic (PLEG): container finished" podID="877b7cf3-5ac1-4e97-b2bc-b0a3129e2016" containerID="2b3af4fe5cc94ba701578d8a6a2a8608eea80bba8d3ece93119061631977e453" exitCode=0 Oct 29 23:03:51.163244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:51.163008 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" event={"ID":"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016","Type":"ContainerDied","Data":"2b3af4fe5cc94ba701578d8a6a2a8608eea80bba8d3ece93119061631977e453"} Oct 29 23:03:52.421563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.421535 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:52.469791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.469759 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-kube-api-access-58cpc\" (UniqueName: \"kubernetes.io/projected/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-kube-api-access-58cpc\") pod \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " Oct 29 23:03:52.469969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.469826 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-openshift-ca\") pod \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " Oct 29 23:03:52.469969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.469869 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-xks-ca\") pod \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\" (UID: \"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016\") " Oct 29 23:03:52.470776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.470746 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-openshift-ca" pod "877b7cf3-5ac1-4e97-b2bc-b0a3129e2016" (UID: "877b7cf3-5ac1-4e97-b2bc-b0a3129e2016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:52.479453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.479420 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-xks-ca" pod "877b7cf3-5ac1-4e97-b2bc-b0a3129e2016" (UID: "877b7cf3-5ac1-4e97-b2bc-b0a3129e2016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:52.479625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.479445 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-kube-api-access-58cpc" pod "877b7cf3-5ac1-4e97-b2bc-b0a3129e2016" (UID: "877b7cf3-5ac1-4e97-b2bc-b0a3129e2016"). InnerVolumeSpecName "kube-api-access-58cpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:52.572966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.572930 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58cpc\" (UniqueName: \"kubernetes.io/projected/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-kube-api-access-58cpc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:52.572966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.572964 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:52.573201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:52.572980 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.182782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:53.182753 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" Oct 29 23:03:53.182965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:53.182784 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr" event={"ID":"877b7cf3-5ac1-4e97-b2bc-b0a3129e2016","Type":"ContainerDied","Data":"c90ce4c9e20c7a223245d2ffb39be3f2aae69a16ca4d39a3d573066f1320c3eb"} Oct 29 23:03:53.182965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:53.182844 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c90ce4c9e20c7a223245d2ffb39be3f2aae69a16ca4d39a3d573066f1320c3eb" Oct 29 23:03:56.833905 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:03:56.833863 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:03:57.032939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:57.032903 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w"] Oct 29 23:03:57.044204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:57.044166 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53592152-tzl4w"] Oct 29 23:03:59.016267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:59.016227 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6660ee6a-5ece-4ddf-9e35-50823d1e25ce" path="/var/lib/kubelet/pods/6660ee6a-5ece-4ddf-9e35-50823d1e25ce/volumes" Oct 29 23:03:59.030312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:59.030277 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7"] Oct 29 23:03:59.064709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:59.064663 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b259700-t6rq7"] Oct 29 23:03:59.777313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:59.777280 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7"] Oct 29 23:03:59.819342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:03:59.819308 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9"] Oct 29 23:04:01.006110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.006073 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21843ebd-cef9-4fd1-9d4e-2d06b1b32a54" path="/var/lib/kubelet/pods/21843ebd-cef9-4fd1-9d4e-2d06b1b32a54/volumes" Oct 29 23:04:01.654825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.654790 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t"] Oct 29 23:04:01.655480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.655465 5736 cpu_manager.go:414] "containerMap: removing container" podUID="877b7cf3-5ac1-4e97-b2bc-b0a3129e2016" containerName="revoke-keys" Oct 29 23:04:01.655532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.655482 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877b7cf3-5ac1-4e97-b2bc-b0a3129e2016" containerName="revoke-keys" Oct 29 23:04:01.655656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.655646 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="877b7cf3-5ac1-4e97-b2bc-b0a3129e2016" containerName="revoke-keys" Oct 29 23:04:01.660676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.660639 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.679142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.679118 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t"] Oct 29 23:04:01.686473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.686450 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:01.686701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.686504 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:04:01.729675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.729623 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hft9g\" (UniqueName: \"kubernetes.io/projected/78597dba-9df1-426d-b0bc-55665c8b3e33-kube-api-access-hft9g\") pod \"maas-api-revoke-keys-e2e-ipp-a-475876-whz6t\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.729855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.729693 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-475876-whz6t\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.730162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.730131 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-475876-whz6t\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.839782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.837344 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-475876-whz6t\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.839782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.837518 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hft9g\" (UniqueName: \"kubernetes.io/projected/78597dba-9df1-426d-b0bc-55665c8b3e33-kube-api-access-hft9g\") pod \"maas-api-revoke-keys-e2e-ipp-a-475876-whz6t\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.839782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.837567 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-475876-whz6t\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.841561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.841533 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-475876-whz6t\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.868024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.867975 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-475876-whz6t\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.871453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.871417 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hft9g\" (UniqueName: \"kubernetes.io/projected/78597dba-9df1-426d-b0bc-55665c8b3e33-kube-api-access-hft9g\") pod \"maas-api-revoke-keys-e2e-ipp-a-475876-whz6t\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:01.974635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:01.974595 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:02.287402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:02.287374 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t"] Oct 29 23:04:02.290080 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:04:02.290048 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78597dba_9df1_426d_b0bc_55665c8b3e33.slice/crio-2d041e9f4a4cdc6ffc5baee81bb7ef17abfd5a38270584fd13ac631cc3e6d86c WatchSource:0}: Error finding container 2d041e9f4a4cdc6ffc5baee81bb7ef17abfd5a38270584fd13ac631cc3e6d86c: Status 404 returned error can't find the container with id 2d041e9f4a4cdc6ffc5baee81bb7ef17abfd5a38270584fd13ac631cc3e6d86c Oct 29 23:04:03.263708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:03.263673 5736 generic.go:356] "Generic (PLEG): container finished" podID="78597dba-9df1-426d-b0bc-55665c8b3e33" containerID="9346fa8b2a901a66574894c142dc73b7c4f1729fa4a545e99d8941c627a3c973" exitCode=0 Oct 29 23:04:03.263865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:03.263732 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" event={"ID":"78597dba-9df1-426d-b0bc-55665c8b3e33","Type":"ContainerDied","Data":"9346fa8b2a901a66574894c142dc73b7c4f1729fa4a545e99d8941c627a3c973"} Oct 29 23:04:03.263865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:03.263780 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" event={"ID":"78597dba-9df1-426d-b0bc-55665c8b3e33","Type":"ContainerStarted","Data":"2d041e9f4a4cdc6ffc5baee81bb7ef17abfd5a38270584fd13ac631cc3e6d86c"} Oct 29 23:04:04.458493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.458466 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:04.585690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.585573 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78597dba-9df1-426d-b0bc-55665c8b3e33-kube-api-access-hft9g\" (UniqueName: \"kubernetes.io/projected/78597dba-9df1-426d-b0bc-55665c8b3e33-kube-api-access-hft9g\") pod \"78597dba-9df1-426d-b0bc-55665c8b3e33\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " Oct 29 23:04:04.585690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.585671 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-xks-ca\") pod \"78597dba-9df1-426d-b0bc-55665c8b3e33\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " Oct 29 23:04:04.585922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.585828 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-openshift-ca\") pod \"78597dba-9df1-426d-b0bc-55665c8b3e33\" (UID: \"78597dba-9df1-426d-b0bc-55665c8b3e33\") " Oct 29 23:04:04.586234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.586210 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-openshift-ca" pod "78597dba-9df1-426d-b0bc-55665c8b3e33" (UID: "78597dba-9df1-426d-b0bc-55665c8b3e33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:04.586601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.586570 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:04.587770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.587743 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-xks-ca" pod "78597dba-9df1-426d-b0bc-55665c8b3e33" (UID: "78597dba-9df1-426d-b0bc-55665c8b3e33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:04.587770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.587742 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78597dba-9df1-426d-b0bc-55665c8b3e33-kube-api-access-hft9g" pod "78597dba-9df1-426d-b0bc-55665c8b3e33" (UID: "78597dba-9df1-426d-b0bc-55665c8b3e33"). InnerVolumeSpecName "kube-api-access-hft9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:04.688422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.688390 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hft9g\" (UniqueName: \"kubernetes.io/projected/78597dba-9df1-426d-b0bc-55665c8b3e33-kube-api-access-hft9g\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:04.688422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.688420 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78597dba-9df1-426d-b0bc-55665c8b3e33-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:04.777757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.777720 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" podUID="a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b" containerName="payload-processing" containerID="cri-o://1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898" gracePeriod=25 Oct 29 23:04:04.820939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:04.820907 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" podUID="ae4c0a47-9162-4e33-af06-4de03e575b45" containerName="payload-pre-processing" containerID="cri-o://cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a" gracePeriod=25 Oct 29 23:04:05.092156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.092110 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:04:05.156209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.156183 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:04:05.210121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.210085 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4c0a47-9162-4e33-af06-4de03e575b45-kube-api-access-w6rld\" (UniqueName: \"kubernetes.io/projected/ae4c0a47-9162-4e33-af06-4de03e575b45-kube-api-access-w6rld\") pod \"ae4c0a47-9162-4e33-af06-4de03e575b45\" (UID: \"ae4c0a47-9162-4e33-af06-4de03e575b45\") " Oct 29 23:04:05.210314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.210292 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-plugins-config-volume\") pod \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\" (UID: \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\") " Oct 29 23:04:05.210429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.210412 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-kube-api-access-8pzqz\" (UniqueName: \"kubernetes.io/projected/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-kube-api-access-8pzqz\") pod \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\" (UID: \"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b\") " Oct 29 23:04:05.210559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.210541 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4c0a47-9162-4e33-af06-4de03e575b45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4c0a47-9162-4e33-af06-4de03e575b45-plugins-config-volume\") pod \"ae4c0a47-9162-4e33-af06-4de03e575b45\" (UID: \"ae4c0a47-9162-4e33-af06-4de03e575b45\") " Oct 29 23:04:05.210722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.210693 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-plugins-config-volume" pod "a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b" (UID: "a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:05.211113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.211087 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4c0a47-9162-4e33-af06-4de03e575b45-plugins-config-volume" pod "ae4c0a47-9162-4e33-af06-4de03e575b45" (UID: "ae4c0a47-9162-4e33-af06-4de03e575b45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:05.211459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.211429 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4c0a47-9162-4e33-af06-4de03e575b45-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:05.211459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.211456 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:05.212790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.212766 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4c0a47-9162-4e33-af06-4de03e575b45-kube-api-access-w6rld" pod "ae4c0a47-9162-4e33-af06-4de03e575b45" (UID: "ae4c0a47-9162-4e33-af06-4de03e575b45"). InnerVolumeSpecName "kube-api-access-w6rld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:05.212962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.212944 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-kube-api-access-8pzqz" pod "a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b" (UID: "a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b"). InnerVolumeSpecName "kube-api-access-8pzqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:05.280759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.280724 5736 generic.go:356] "Generic (PLEG): container finished" podID="a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b" containerID="1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898" exitCode=0 Oct 29 23:04:05.280895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.280776 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" Oct 29 23:04:05.280895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.280805 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" event={"ID":"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b","Type":"ContainerDied","Data":"1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898"} Oct 29 23:04:05.280895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.280863 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7" event={"ID":"a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b","Type":"ContainerDied","Data":"d1bc60a7fc2e65156504428ec937bcee4595d3b394a06a703a6d9127acf6cc45"} Oct 29 23:04:05.280895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.280887 5736 scope.go:122] "RemoveContainer" containerID="1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898" Oct 29 23:04:05.282381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.282359 5736 generic.go:356] "Generic (PLEG): container finished" podID="ae4c0a47-9162-4e33-af06-4de03e575b45" containerID="cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a" exitCode=0 Oct 29 23:04:05.282460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.282389 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" event={"ID":"ae4c0a47-9162-4e33-af06-4de03e575b45","Type":"ContainerDied","Data":"cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a"} Oct 29 23:04:05.282460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.282426 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" Oct 29 23:04:05.282460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.282440 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9" event={"ID":"ae4c0a47-9162-4e33-af06-4de03e575b45","Type":"ContainerDied","Data":"70a717cb09345fb5d33c0f035c897e60405f2fcbaa1288bc834f0c28fa3e6f2a"} Oct 29 23:04:05.284061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.284040 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" event={"ID":"78597dba-9df1-426d-b0bc-55665c8b3e33","Type":"ContainerDied","Data":"2d041e9f4a4cdc6ffc5baee81bb7ef17abfd5a38270584fd13ac631cc3e6d86c"} Oct 29 23:04:05.284189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.284071 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d041e9f4a4cdc6ffc5baee81bb7ef17abfd5a38270584fd13ac631cc3e6d86c" Oct 29 23:04:05.284189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.284082 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t" Oct 29 23:04:05.293453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.293435 5736 scope.go:122] "RemoveContainer" containerID="1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898" Oct 29 23:04:05.293777 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:05.293755 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898\": container with ID starting with 1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898 not found: ID does not exist" containerID="1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898" Oct 29 23:04:05.293869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.293782 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898"} err="failed to get container status \"1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898\": rpc error: code = NotFound desc = could not find container \"1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898\": container with ID starting with 1133104f430edd0928197f5520723cd6d7907fbc0c97489a48399650abc60898 not found: ID does not exist" Oct 29 23:04:05.293869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.293799 5736 scope.go:122] "RemoveContainer" containerID="cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a" Oct 29 23:04:05.303441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.303424 5736 scope.go:122] "RemoveContainer" containerID="cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a" Oct 29 23:04:05.303730 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:05.303709 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a\": container with ID starting with cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a not found: ID does not exist" containerID="cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a" Oct 29 23:04:05.303796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.303736 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a"} err="failed to get container status \"cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a\": rpc error: code = NotFound desc = could not find container \"cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a\": container with ID starting with cbee0501416605ce0ea5d835c6ac187bc17629eb1dadc8ed5085b9ae760aa84a not found: ID does not exist" Oct 29 23:04:05.312892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.312870 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pzqz\" (UniqueName: \"kubernetes.io/projected/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b-kube-api-access-8pzqz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:05.312956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.312893 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rld\" (UniqueName: \"kubernetes.io/projected/ae4c0a47-9162-4e33-af06-4de03e575b45-kube-api-access-w6rld\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:05.342802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.340865 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9"] Oct 29 23:04:05.342947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.342871 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c486c-65dc84bb9f-22sq9"] Oct 29 23:04:05.371654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.371633 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7"] Oct 29 23:04:05.375410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:05.375391 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c486c-664c6498f4-mgqh7"] Oct 29 23:04:07.043742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:07.043687 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b" path="/var/lib/kubelet/pods/a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b/volumes" Oct 29 23:04:07.044317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:07.044292 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4c0a47-9162-4e33-af06-4de03e575b45" path="/var/lib/kubelet/pods/ae4c0a47-9162-4e33-af06-4de03e575b45/volumes" Oct 29 23:04:08.150471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:08.150418 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp"] Oct 29 23:04:08.163648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:08.163571 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-660540-hk5mp"] Oct 29 23:04:09.005907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:09.005876 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543f43db-04e2-4d24-a8cc-97815e21a22c" path="/var/lib/kubelet/pods/543f43db-04e2-4d24-a8cc-97815e21a22c/volumes" Oct 29 23:04:10.031837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.030832 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc"] Oct 29 23:04:10.042017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.041984 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba8d575-7b2xc"] Oct 29 23:04:10.964674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.964597 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2"] Oct 29 23:04:10.965748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.965722 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b" containerName="payload-processing" Oct 29 23:04:10.965748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.965748 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b" containerName="payload-processing" Oct 29 23:04:10.965933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.965782 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ae4c0a47-9162-4e33-af06-4de03e575b45" containerName="payload-pre-processing" Oct 29 23:04:10.965933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.965792 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4c0a47-9162-4e33-af06-4de03e575b45" containerName="payload-pre-processing" Oct 29 23:04:10.965933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.965828 5736 cpu_manager.go:414] "containerMap: removing container" podUID="78597dba-9df1-426d-b0bc-55665c8b3e33" containerName="revoke-keys" Oct 29 23:04:10.965933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.965836 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78597dba-9df1-426d-b0bc-55665c8b3e33" containerName="revoke-keys" Oct 29 23:04:10.966140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.966025 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09a1e09-2ee5-4a80-bb8b-d0c020a2cd1b" containerName="payload-processing" Oct 29 23:04:10.966140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.966038 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4c0a47-9162-4e33-af06-4de03e575b45" containerName="payload-pre-processing" Oct 29 23:04:10.966140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.966066 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="78597dba-9df1-426d-b0bc-55665c8b3e33" containerName="revoke-keys" Oct 29 23:04:10.969893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.969873 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:10.989110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:10.989079 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2"] Oct 29 23:04:11.015702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.015666 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad24946-8dd9-4340-93d9-3abb9ca312bf" path="/var/lib/kubelet/pods/8ad24946-8dd9-4340-93d9-3abb9ca312bf/volumes" Oct 29 23:04:11.159593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.159540 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q72b\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-kube-api-access-8q72b\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.160097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.159630 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.160097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.159781 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.160097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.159813 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83afa120-ca9b-4466-9323-1668b74fbd48-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.160097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.159876 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.160097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.159892 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.160097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.159919 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.160097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.159948 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83afa120-ca9b-4466-9323-1668b74fbd48-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.160097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.159998 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.260971 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261007 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261040 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261082 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83afa120-ca9b-4466-9323-1668b74fbd48-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261144 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261300 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q72b\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-kube-api-access-8q72b\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261362 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261444 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261509 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261522 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261546 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261623 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83afa120-ca9b-4466-9323-1668b74fbd48-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.261811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.261787 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.262352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.262330 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83afa120-ca9b-4466-9323-1668b74fbd48-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.263654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.263634 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.263852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.263832 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83afa120-ca9b-4466-9323-1668b74fbd48-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.284532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.284509 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.286690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.286669 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q72b\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-kube-api-access-8q72b\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.292575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.292553 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:11.638107 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:04:11.638073 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83afa120_ca9b_4466_9323_1668b74fbd48.slice/crio-72059ed3e1e0366d274c3d164a63eee97642d6b0fe6d127eaa934b6e285afc78 WatchSource:0}: Error finding container 72059ed3e1e0366d274c3d164a63eee97642d6b0fe6d127eaa934b6e285afc78: Status 404 returned error can't find the container with id 72059ed3e1e0366d274c3d164a63eee97642d6b0fe6d127eaa934b6e285afc78 Oct 29 23:04:11.638477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.638451 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2"] Oct 29 23:04:11.641305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.641269 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:04:11.641382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.641354 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:04:11.641421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:11.641399 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:04:12.416973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:12.416933 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" event={"ID":"83afa120-ca9b-4466-9323-1668b74fbd48","Type":"ContainerStarted","Data":"a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03"} Oct 29 23:04:12.417384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:12.417004 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" event={"ID":"83afa120-ca9b-4466-9323-1668b74fbd48","Type":"ContainerStarted","Data":"72059ed3e1e0366d274c3d164a63eee97642d6b0fe6d127eaa934b6e285afc78"} Oct 29 23:04:12.491253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:12.491195 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" podStartSLOduration=2.491174996 podStartE2EDuration="2.491174996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:12.479669435 +0000 UTC m=+1249.595589040" watchObservedRunningTime="2026-10-29 23:04:12.491174996 +0000 UTC m=+1249.607094608" Oct 29 23:04:13.292740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:13.292706 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:13.294616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:13.294565 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:13.294739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:13.294628 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 29 23:04:14.293466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:14.293435 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:14.293972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:14.293502 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 29 23:04:15.293496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:15.293464 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:15.293909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:15.293530 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 29 23:04:16.280301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.280257 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs"] Oct 29 23:04:16.283123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.283082 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" containerID="cri-o://02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032" gracePeriod=30 Oct 29 23:04:16.287971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.287939 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp"] Oct 29 23:04:16.288546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.288494 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" containerID="cri-o://917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9" gracePeriod=30 Oct 29 23:04:16.294091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.294065 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml"] Oct 29 23:04:16.294488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.294255 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" containerID="cri-o://3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718" gracePeriod=30 Oct 29 23:04:16.294488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.294422 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:16.294488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.294450 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 29 23:04:16.861591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.861537 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt"] Oct 29 23:04:16.866441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.866417 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:16.883867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.883837 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt"] Oct 29 23:04:16.912194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.912162 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:04:16.912379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.912245 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:16.978419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.978380 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gx6\" (UniqueName: \"kubernetes.io/projected/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-kube-api-access-j4gx6\") pod \"maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:16.978680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.978655 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:16.978864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:16.978844 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:17.080109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.080069 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:17.080278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.080213 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:17.080364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.080347 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gx6\" (UniqueName: \"kubernetes.io/projected/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-kube-api-access-j4gx6\") pod \"maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:17.080946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.080923 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:17.082668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.082633 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:17.117163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.117089 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gx6\" (UniqueName: \"kubernetes.io/projected/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-kube-api-access-j4gx6\") pod \"maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:17.181999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.181960 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:17.294561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.294525 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:17.295040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.294614 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 29 23:04:17.604603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:17.579440 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt"] Oct 29 23:04:18.294012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:18.293975 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 29 23:04:18.294187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:18.294030 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 29 23:04:18.697478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:18.697445 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" event={"ID":"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad","Type":"ContainerStarted","Data":"3bc866bf4b15c48cc69f16e78275b6473cc44b01cc87e4f3101debc511016f0f"} Oct 29 23:04:18.697962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:18.697505 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" event={"ID":"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad","Type":"ContainerStarted","Data":"c70b63ca13ecaae559ff0e887dd045a87cd3428d90961d83bfdf44c53eeba31d"} Oct 29 23:04:18.740502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:18.740451 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" podStartSLOduration=2.740437095 podStartE2EDuration="2.740437095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:18.737083446 +0000 UTC m=+1255.853003055" watchObservedRunningTime="2026-10-29 23:04:18.740437095 +0000 UTC m=+1255.856356749" Oct 29 23:04:19.297986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:19.297952 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:19.298382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:19.298352 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:19.299280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:19.299261 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:04:19.710416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:19.710372 5736 generic.go:356] "Generic (PLEG): container finished" podID="ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad" containerID="3bc866bf4b15c48cc69f16e78275b6473cc44b01cc87e4f3101debc511016f0f" exitCode=0 Oct 29 23:04:19.711692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:19.711663 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" event={"ID":"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad","Type":"ContainerDied","Data":"3bc866bf4b15c48cc69f16e78275b6473cc44b01cc87e4f3101debc511016f0f"} Oct 29 23:04:20.880596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:20.880555 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:20.984889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:20.984807 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-openshift-ca\") pod \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " Oct 29 23:04:20.984889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:20.984849 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-xks-ca\") pod \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " Oct 29 23:04:20.985128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:20.985009 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-kube-api-access-j4gx6\" (UniqueName: \"kubernetes.io/projected/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-kube-api-access-j4gx6\") pod \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\" (UID: \"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad\") " Oct 29 23:04:20.985244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:20.985220 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-openshift-ca" pod "ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad" (UID: "ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:20.985807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:20.985779 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:20.987678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:20.987648 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-kube-api-access-j4gx6" pod "ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad" (UID: "ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad"). InnerVolumeSpecName "kube-api-access-j4gx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:20.987802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:20.987763 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-xks-ca" pod "ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad" (UID: "ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:21.088002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.087970 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gx6\" (UniqueName: \"kubernetes.io/projected/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-kube-api-access-j4gx6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.088002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.088004 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.566251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.566222 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:04:21.625637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.625267 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:04:21.681646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.681620 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:04:21.705001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.704968 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-token\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-token\") pod \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " Oct 29 23:04:21.705179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705020 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istiod-ca-cert\") pod \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " Oct 29 23:04:21.705179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705111 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-certs\") pod \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " Oct 29 23:04:21.705179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705168 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-kube-api-access-k9279\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-kube-api-access-k9279\") pod \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705188 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-credential-socket\") pod \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705211 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istiod-ca-cert\") pod \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705230 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-token\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-token\") pod \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705278 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-credential-socket\") pod \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705314 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-data\") pod \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705348 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-kube-api-access-89qk9\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-kube-api-access-89qk9\") pod \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705392 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-envoy\") pod \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705444 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-podinfo\") pod \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705502 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-socket\") pod \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\" (UID: \"811433d3-39ab-4d7f-9c9e-baa41b8b025b\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705663 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-data\") pod \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " Oct 29 23:04:21.705952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.705710 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-socket\") pod \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " Oct 29 23:04:21.706638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.706096 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-socket" pod "e37cdb27-0846-40e3-b981-4bcf72fbd83c" (UID: "e37cdb27-0846-40e3-b981-4bcf72fbd83c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.706638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.706468 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-data" pod "811433d3-39ab-4d7f-9c9e-baa41b8b025b" (UID: "811433d3-39ab-4d7f-9c9e-baa41b8b025b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.706914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.706883 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-credential-socket" pod "811433d3-39ab-4d7f-9c9e-baa41b8b025b" (UID: "811433d3-39ab-4d7f-9c9e-baa41b8b025b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.707235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.707207 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-certs" pod "811433d3-39ab-4d7f-9c9e-baa41b8b025b" (UID: "811433d3-39ab-4d7f-9c9e-baa41b8b025b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.707502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.707478 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-credential-socket" pod "e37cdb27-0846-40e3-b981-4bcf72fbd83c" (UID: "e37cdb27-0846-40e3-b981-4bcf72fbd83c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.707615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.707563 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-data" pod "e37cdb27-0846-40e3-b981-4bcf72fbd83c" (UID: "e37cdb27-0846-40e3-b981-4bcf72fbd83c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.708229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.708200 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-socket" pod "811433d3-39ab-4d7f-9c9e-baa41b8b025b" (UID: "811433d3-39ab-4d7f-9c9e-baa41b8b025b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.708429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.708407 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istiod-ca-cert" pod "e37cdb27-0846-40e3-b981-4bcf72fbd83c" (UID: "e37cdb27-0846-40e3-b981-4bcf72fbd83c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:21.708515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.708472 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-podinfo\") pod \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " Oct 29 23:04:21.708609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.708571 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-envoy\") pod \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " Oct 29 23:04:21.708678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.708652 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-certs\") pod \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\" (UID: \"e37cdb27-0846-40e3-b981-4bcf72fbd83c\") " Oct 29 23:04:21.708893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.708868 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-kube-api-access-k9279" pod "811433d3-39ab-4d7f-9c9e-baa41b8b025b" (UID: "811433d3-39ab-4d7f-9c9e-baa41b8b025b"). InnerVolumeSpecName "kube-api-access-k9279". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.709728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.709701 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-envoy" pod "811433d3-39ab-4d7f-9c9e-baa41b8b025b" (UID: "811433d3-39ab-4d7f-9c9e-baa41b8b025b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.710311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.710273 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istiod-ca-cert" pod "811433d3-39ab-4d7f-9c9e-baa41b8b025b" (UID: "811433d3-39ab-4d7f-9c9e-baa41b8b025b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:21.710769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.710736 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-certs" pod "e37cdb27-0846-40e3-b981-4bcf72fbd83c" (UID: "e37cdb27-0846-40e3-b981-4bcf72fbd83c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.713326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.713287 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-token" pod "811433d3-39ab-4d7f-9c9e-baa41b8b025b" (UID: "811433d3-39ab-4d7f-9c9e-baa41b8b025b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.713549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.713523 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-podinfo" pod "811433d3-39ab-4d7f-9c9e-baa41b8b025b" (UID: "811433d3-39ab-4d7f-9c9e-baa41b8b025b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:21.713716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.713622 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-envoy" pod "e37cdb27-0846-40e3-b981-4bcf72fbd83c" (UID: "e37cdb27-0846-40e3-b981-4bcf72fbd83c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.714707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714678 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.714707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714704 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9279\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-kube-api-access-k9279\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.714851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714719 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.714851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714731 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.714851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714744 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.714851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714745 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-token" pod "e37cdb27-0846-40e3-b981-4bcf72fbd83c" (UID: "e37cdb27-0846-40e3-b981-4bcf72fbd83c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.714851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714759 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.714851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714772 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.714851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714786 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.714851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.714799 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.715306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.715281 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-podinfo" pod "e37cdb27-0846-40e3-b981-4bcf72fbd83c" (UID: "e37cdb27-0846-40e3-b981-4bcf72fbd83c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:21.715415 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.715391 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-kube-api-access-89qk9" pod "e37cdb27-0846-40e3-b981-4bcf72fbd83c" (UID: "e37cdb27-0846-40e3-b981-4bcf72fbd83c"). InnerVolumeSpecName "kube-api-access-89qk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.735073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.733348 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" event={"ID":"ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad","Type":"ContainerDied","Data":"c70b63ca13ecaae559ff0e887dd045a87cd3428d90961d83bfdf44c53eeba31d"} Oct 29 23:04:21.735073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.733406 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c70b63ca13ecaae559ff0e887dd045a87cd3428d90961d83bfdf44c53eeba31d" Oct 29 23:04:21.735073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.733479 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt" Oct 29 23:04:21.740667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.737165 5736 generic.go:356] "Generic (PLEG): container finished" podID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerID="917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9" exitCode=0 Oct 29 23:04:21.740667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.737238 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" event={"ID":"811433d3-39ab-4d7f-9c9e-baa41b8b025b","Type":"ContainerDied","Data":"917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9"} Oct 29 23:04:21.740667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.737290 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" event={"ID":"811433d3-39ab-4d7f-9c9e-baa41b8b025b","Type":"ContainerDied","Data":"bf84ea9109640b6c9e21c3814fae7bba8904c76568818ebcce675008ccd9f941"} Oct 29 23:04:21.740667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.737321 5736 scope.go:122] "RemoveContainer" containerID="917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9" Oct 29 23:04:21.740667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.737468 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp" Oct 29 23:04:21.741639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.740719 5736 generic.go:356] "Generic (PLEG): container finished" podID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerID="3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718" exitCode=0 Oct 29 23:04:21.741639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.740794 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" event={"ID":"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2","Type":"ContainerDied","Data":"3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718"} Oct 29 23:04:21.741639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.740830 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" event={"ID":"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2","Type":"ContainerDied","Data":"6cd242bc4350de1ec0de12c8f741bc2e1166e69ee15126bdb047027c0a6c7287"} Oct 29 23:04:21.741639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.740902 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml" Oct 29 23:04:21.744416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.744382 5736 generic.go:356] "Generic (PLEG): container finished" podID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerID="02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032" exitCode=0 Oct 29 23:04:21.744522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.744437 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" event={"ID":"e37cdb27-0846-40e3-b981-4bcf72fbd83c","Type":"ContainerDied","Data":"02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032"} Oct 29 23:04:21.744522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.744472 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" event={"ID":"e37cdb27-0846-40e3-b981-4bcf72fbd83c","Type":"ContainerDied","Data":"09390b959d5b8bcf4f6fe7e0b5dc1a1fc483940aaf4b7ab5e7653d0f9d91f8b6"} Oct 29 23:04:21.744651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.744530 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs" Oct 29 23:04:21.750979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.750957 5736 scope.go:122] "RemoveContainer" containerID="917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9" Oct 29 23:04:21.751358 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:21.751331 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9\": container with ID starting with 917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9 not found: ID does not exist" containerID="917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9" Oct 29 23:04:21.751443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.751364 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9"} err="failed to get container status \"917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9\": rpc error: code = NotFound desc = could not find container \"917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9\": container with ID starting with 917e54380203ef4a3fc5ca0e69c1825006981b3381939f457847a4b2f6aa4fd9 not found: ID does not exist" Oct 29 23:04:21.751443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.751382 5736 scope.go:122] "RemoveContainer" containerID="3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718" Oct 29 23:04:21.768038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.768020 5736 scope.go:122] "RemoveContainer" containerID="3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718" Oct 29 23:04:21.768388 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:21.768363 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718\": container with ID starting with 3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718 not found: ID does not exist" containerID="3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718" Oct 29 23:04:21.768480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.768399 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718"} err="failed to get container status \"3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718\": rpc error: code = NotFound desc = could not find container \"3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718\": container with ID starting with 3cc1c6a6da6cab2f89251e624f3b73f17af0e053794db0a14ae2e0b0a826d718 not found: ID does not exist" Oct 29 23:04:21.768480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.768422 5736 scope.go:122] "RemoveContainer" containerID="02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032" Oct 29 23:04:21.787272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.787235 5736 scope.go:122] "RemoveContainer" containerID="02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032" Oct 29 23:04:21.788146 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:21.788118 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032\": container with ID starting with 02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032 not found: ID does not exist" containerID="02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032" Oct 29 23:04:21.788293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.788155 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032"} err="failed to get container status \"02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032\": rpc error: code = NotFound desc = could not find container \"02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032\": container with ID starting with 02d8b5203dd8c2532d1eb5518ffb886cafdb8fbfc0f2db3fb128326fdc7e6032 not found: ID does not exist" Oct 29 23:04:21.816280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816211 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istiod-ca-cert\") pod \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " Oct 29 23:04:21.816280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816265 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-certs\") pod \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " Oct 29 23:04:21.816460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816332 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-data\") pod \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " Oct 29 23:04:21.816524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816503 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-envoy\") pod \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " Oct 29 23:04:21.816626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816540 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-token\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-token\") pod \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " Oct 29 23:04:21.816626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816591 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-podinfo\") pod \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " Oct 29 23:04:21.816738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816688 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-credential-socket\") pod \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " Oct 29 23:04:21.816796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816760 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-kube-api-access-qdc52\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-kube-api-access-qdc52\") pod \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " Oct 29 23:04:21.816850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816801 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-socket\") pod \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\" (UID: \"80ce37c3-b937-4ba0-bd43-02ad0a93ecc2\") " Oct 29 23:04:21.816952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.816924 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-data" pod "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" (UID: "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.817303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.817277 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istiod-ca-cert" pod "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" (UID: "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:21.818210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.817516 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-certs" pod "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" (UID: "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.818775 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-credential-socket" pod "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" (UID: "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.818938 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-socket" pod "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" (UID: "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819160 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819179 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819193 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819207 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819220 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819233 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819246 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qk9\" (UniqueName: \"kubernetes.io/projected/e37cdb27-0846-40e3-b981-4bcf72fbd83c-kube-api-access-89qk9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819260 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819273 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819286 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819298 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/811433d3-39ab-4d7f-9c9e-baa41b8b025b-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819310 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819322 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.820884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.819337 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37cdb27-0846-40e3-b981-4bcf72fbd83c-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.823980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.823910 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-token" pod "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" (UID: "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.824245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.824221 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j"] Oct 29 23:04:21.825628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.825604 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad" containerName="revoke-keys" Oct 29 23:04:21.825628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.825629 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad" containerName="revoke-keys" Oct 29 23:04:21.825800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.825657 5736 cpu_manager.go:414] "containerMap: removing container" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" Oct 29 23:04:21.825800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.825665 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" Oct 29 23:04:21.825800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.825682 5736 cpu_manager.go:414] "containerMap: removing container" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" Oct 29 23:04:21.825800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.825689 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" Oct 29 23:04:21.825800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.825702 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" Oct 29 23:04:21.825800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.825710 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" Oct 29 23:04:21.826355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.826070 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad" containerName="revoke-keys" Oct 29 23:04:21.826355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.826095 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" containerName="istio-proxy" Oct 29 23:04:21.826355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.826106 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" containerName="istio-proxy" Oct 29 23:04:21.826355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.826120 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" containerName="istio-proxy" Oct 29 23:04:21.828468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.828441 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-podinfo" pod "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" (UID: "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:21.828593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.828560 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-envoy" pod "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" (UID: "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:21.830972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.830940 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-kube-api-access-qdc52" pod "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" (UID: "80ce37c3-b937-4ba0-bd43-02ad0a93ecc2"). InnerVolumeSpecName "kube-api-access-qdc52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:21.832039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.832020 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:21.873978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.854406 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j"] Oct 29 23:04:21.873978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.860763 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs"] Oct 29 23:04:21.873978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.867040 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-28frs"] Oct 29 23:04:21.907907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.899348 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp"] Oct 29 23:04:21.907907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.904873 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-wkdqp"] Oct 29 23:04:21.921466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.921442 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.921466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.921465 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.921646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.921477 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:21.921646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:21.921486 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdc52\" (UniqueName: \"kubernetes.io/projected/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2-kube-api-access-qdc52\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:22.022680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.022639 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.022862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.022760 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452b7640-c7d1-4abf-9ade-695d97f29762-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.022862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.022815 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.022862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.022852 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.022988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.022975 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452b7640-c7d1-4abf-9ade-695d97f29762-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.023050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.023037 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.023169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.023128 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwwr4\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-kube-api-access-rwwr4\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.023279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.023233 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.023340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.023327 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.130534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.130496 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.130873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.130850 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.130958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.130947 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452b7640-c7d1-4abf-9ade-695d97f29762-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.131034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.131017 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.131097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.131082 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.131156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.131129 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452b7640-c7d1-4abf-9ade-695d97f29762-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.131230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.131217 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.131303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.131260 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwwr4\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-kube-api-access-rwwr4\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.131390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.131345 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.131803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.131779 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.132120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.132091 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.132249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.132230 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.133006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.132983 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452b7640-c7d1-4abf-9ade-695d97f29762-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.133232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.133213 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.133652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.133568 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml"] Oct 29 23:04:22.134123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.134091 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-nq6ml"] Oct 29 23:04:22.135474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.135448 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452b7640-c7d1-4abf-9ade-695d97f29762-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.135566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.135454 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.154932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.154901 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.155830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.155809 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwwr4\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-kube-api-access-rwwr4\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.199129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.199096 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:22.527853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.527826 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j"] Oct 29 23:04:22.530203 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:04:22.530172 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452b7640_c7d1_4abf_9ade_695d97f29762.slice/crio-fb3f92fc1c43eccc2d3c831a45ebe0c9ba6f026940b29ce26ad7da72f9b56567 WatchSource:0}: Error finding container fb3f92fc1c43eccc2d3c831a45ebe0c9ba6f026940b29ce26ad7da72f9b56567: Status 404 returned error can't find the container with id fb3f92fc1c43eccc2d3c831a45ebe0c9ba6f026940b29ce26ad7da72f9b56567 Oct 29 23:04:22.533066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.533023 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:04:22.533182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.533116 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:04:22.533182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.533172 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:04:22.759185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.759150 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" event={"ID":"452b7640-c7d1-4abf-9ade-695d97f29762","Type":"ContainerStarted","Data":"a1f7d022077682f46939a91f38274004b7837495f56bceb79e18708d70d356ed"} Oct 29 23:04:22.759387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.759201 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" event={"ID":"452b7640-c7d1-4abf-9ade-695d97f29762","Type":"ContainerStarted","Data":"fb3f92fc1c43eccc2d3c831a45ebe0c9ba6f026940b29ce26ad7da72f9b56567"} Oct 29 23:04:22.805570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:22.805415 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" podStartSLOduration=1.805398147 podStartE2EDuration="1.805398147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:22.803366364 +0000 UTC m=+1259.919285970" watchObservedRunningTime="2026-10-29 23:04:22.805398147 +0000 UTC m=+1259.921317754" Oct 29 23:04:23.014802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:23.014769 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ce37c3-b937-4ba0-bd43-02ad0a93ecc2" path="/var/lib/kubelet/pods/80ce37c3-b937-4ba0-bd43-02ad0a93ecc2/volumes" Oct 29 23:04:23.015311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:23.015296 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811433d3-39ab-4d7f-9c9e-baa41b8b025b" path="/var/lib/kubelet/pods/811433d3-39ab-4d7f-9c9e-baa41b8b025b/volumes" Oct 29 23:04:23.015807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:23.015792 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37cdb27-0846-40e3-b981-4bcf72fbd83c" path="/var/lib/kubelet/pods/e37cdb27-0846-40e3-b981-4bcf72fbd83c/volumes" Oct 29 23:04:23.200449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:23.200361 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:23.201806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:23.201771 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:23.201893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:23.201830 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 23:04:23.856167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:23.856133 5736 scope.go:122] "RemoveContainer" containerID="5b45747476c4b0d7ebdaf0dabef71c584e720e01f690ef9e0b99c0b5a8c8d803" Oct 29 23:04:24.200875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:24.200789 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:24.200875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:24.200845 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 23:04:25.200704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:25.200670 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:25.200862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:25.200721 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 23:04:26.199849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:26.199817 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:26.200245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:26.199872 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 23:04:26.824691 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:26.824653 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:04:27.200753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:27.200665 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:27.200753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:27.200723 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 23:04:28.199789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:28.199750 5736 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 23:04:28.199964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:28.199816 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 23:04:29.205290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:29.205256 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:29.205791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:29.205770 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:29.206449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:29.206431 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:04:30.272749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.272717 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g"] Oct 29 23:04:30.273263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.272976 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" podUID="d42eb596-4e68-4419-94d7-733d66c152c1" containerName="maas-api" containerID="cri-o://3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c" gracePeriod=30 Oct 29 23:04:30.626287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.625959 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:04:30.704993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.704941 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955"] Oct 29 23:04:30.705855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.705804 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" podUID="d5246614-6bd8-496d-b7e3-34a5ff531e5e" containerName="maas-api" containerID="cri-o://daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115" gracePeriod=30 Oct 29 23:04:30.721234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.721198 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-metrics-certs\") pod \"d42eb596-4e68-4419-94d7-733d66c152c1\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " Oct 29 23:04:30.721842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.721573 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42eb596-4e68-4419-94d7-733d66c152c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42eb596-4e68-4419-94d7-733d66c152c1-openshift-service-ca\") pod \"d42eb596-4e68-4419-94d7-733d66c152c1\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " Oct 29 23:04:30.721842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.721649 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42eb596-4e68-4419-94d7-733d66c152c1-kube-api-access-94cn9\" (UniqueName: \"kubernetes.io/projected/d42eb596-4e68-4419-94d7-733d66c152c1-kube-api-access-94cn9\") pod \"d42eb596-4e68-4419-94d7-733d66c152c1\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " Oct 29 23:04:30.721842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.721725 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls\") pod \"d42eb596-4e68-4419-94d7-733d66c152c1\" (UID: \"d42eb596-4e68-4419-94d7-733d66c152c1\") " Oct 29 23:04:30.724221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.724038 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42eb596-4e68-4419-94d7-733d66c152c1-openshift-service-ca" pod "d42eb596-4e68-4419-94d7-733d66c152c1" (UID: "d42eb596-4e68-4419-94d7-733d66c152c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:30.725974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.725934 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls" pod "d42eb596-4e68-4419-94d7-733d66c152c1" (UID: "d42eb596-4e68-4419-94d7-733d66c152c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:30.734003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.733976 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-metrics-certs" pod "d42eb596-4e68-4419-94d7-733d66c152c1" (UID: "d42eb596-4e68-4419-94d7-733d66c152c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:30.734437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.734412 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42eb596-4e68-4419-94d7-733d66c152c1-kube-api-access-94cn9" pod "d42eb596-4e68-4419-94d7-733d66c152c1" (UID: "d42eb596-4e68-4419-94d7-733d66c152c1"). InnerVolumeSpecName "kube-api-access-94cn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:30.824648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.824550 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42eb596-4e68-4419-94d7-733d66c152c1-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:30.824648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.824604 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94cn9\" (UniqueName: \"kubernetes.io/projected/d42eb596-4e68-4419-94d7-733d66c152c1-kube-api-access-94cn9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:30.824648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.824623 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:30.824648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.824636 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42eb596-4e68-4419-94d7-733d66c152c1-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:30.834759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.834726 5736 generic.go:356] "Generic (PLEG): container finished" podID="d42eb596-4e68-4419-94d7-733d66c152c1" containerID="3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c" exitCode=0 Oct 29 23:04:30.834886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.834820 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" Oct 29 23:04:30.834886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.834846 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" event={"ID":"d42eb596-4e68-4419-94d7-733d66c152c1","Type":"ContainerDied","Data":"3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c"} Oct 29 23:04:30.835003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.834907 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g" event={"ID":"d42eb596-4e68-4419-94d7-733d66c152c1","Type":"ContainerDied","Data":"0e34b385484752f0fceb46425b943f1152872f176e39b86c62644e45de43ce80"} Oct 29 23:04:30.835003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.834940 5736 scope.go:122] "RemoveContainer" containerID="3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c" Oct 29 23:04:30.854449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.854428 5736 scope.go:122] "RemoveContainer" containerID="3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c" Oct 29 23:04:30.854787 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:30.854765 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c\": container with ID starting with 3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c not found: ID does not exist" containerID="3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c" Oct 29 23:04:30.854852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.854795 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c"} err="failed to get container status \"3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c\": rpc error: code = NotFound desc = could not find container \"3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c\": container with ID starting with 3151650d013d8c70685f670c7c14a5339cf679de7f8a3b80bc5c8831de44c84c not found: ID does not exist" Oct 29 23:04:30.896725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.896691 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g"] Oct 29 23:04:30.904005 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:30.903974 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-475876-9f94d9bd7-g2x4g"] Oct 29 23:04:31.007647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.007620 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42eb596-4e68-4419-94d7-733d66c152c1" path="/var/lib/kubelet/pods/d42eb596-4e68-4419-94d7-733d66c152c1/volumes" Oct 29 23:04:31.036404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.036358 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t"] Oct 29 23:04:31.042720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.042687 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-475876-whz6t"] Oct 29 23:04:31.054644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.054622 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:04:31.231661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.206003 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5246614-6bd8-496d-b7e3-34a5ff531e5e-kube-api-access-t52z7\" (UniqueName: \"kubernetes.io/projected/d5246614-6bd8-496d-b7e3-34a5ff531e5e-kube-api-access-t52z7\") pod \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " Oct 29 23:04:31.232101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.232063 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5246614-6bd8-496d-b7e3-34a5ff531e5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5246614-6bd8-496d-b7e3-34a5ff531e5e-openshift-service-ca\") pod \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " Oct 29 23:04:31.232276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.232259 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls\") pod \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " Oct 29 23:04:31.232504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.232488 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-metrics-certs\") pod \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\" (UID: \"d5246614-6bd8-496d-b7e3-34a5ff531e5e\") " Oct 29 23:04:31.245205 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.244938 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5246614-6bd8-496d-b7e3-34a5ff531e5e-kube-api-access-t52z7" pod "d5246614-6bd8-496d-b7e3-34a5ff531e5e" (UID: "d5246614-6bd8-496d-b7e3-34a5ff531e5e"). InnerVolumeSpecName "kube-api-access-t52z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:31.245502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.245479 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5246614-6bd8-496d-b7e3-34a5ff531e5e-openshift-service-ca" pod "d5246614-6bd8-496d-b7e3-34a5ff531e5e" (UID: "d5246614-6bd8-496d-b7e3-34a5ff531e5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:31.348984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.309361 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-metrics-certs" pod "d5246614-6bd8-496d-b7e3-34a5ff531e5e" (UID: "d5246614-6bd8-496d-b7e3-34a5ff531e5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:31.506668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.498640 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls" pod "d5246614-6bd8-496d-b7e3-34a5ff531e5e" (UID: "d5246614-6bd8-496d-b7e3-34a5ff531e5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:31.506838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.506710 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5246614-6bd8-496d-b7e3-34a5ff531e5e-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:31.506838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.506744 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:31.506838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.506758 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5246614-6bd8-496d-b7e3-34a5ff531e5e-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:31.506838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.506770 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52z7\" (UniqueName: \"kubernetes.io/projected/d5246614-6bd8-496d-b7e3-34a5ff531e5e-kube-api-access-t52z7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:31.762465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.762376 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459"] Oct 29 23:04:31.771313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.771277 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec4f6d-vr459"] Oct 29 23:04:31.844346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.844310 5736 generic.go:356] "Generic (PLEG): container finished" podID="d5246614-6bd8-496d-b7e3-34a5ff531e5e" containerID="daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115" exitCode=0 Oct 29 23:04:31.844516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.844374 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" Oct 29 23:04:31.844516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.844390 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" event={"ID":"d5246614-6bd8-496d-b7e3-34a5ff531e5e","Type":"ContainerDied","Data":"daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115"} Oct 29 23:04:31.844516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.844474 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955" event={"ID":"d5246614-6bd8-496d-b7e3-34a5ff531e5e","Type":"ContainerDied","Data":"299a962f942a8431c7b7142f5c783c54d113a2df349bef4bc84a400e59baa213"} Oct 29 23:04:31.844516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.844503 5736 scope.go:122] "RemoveContainer" containerID="daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115" Oct 29 23:04:31.857695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.857662 5736 scope.go:122] "RemoveContainer" containerID="daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115" Oct 29 23:04:31.858021 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:31.857994 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115\": container with ID starting with daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115 not found: ID does not exist" containerID="daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115" Oct 29 23:04:31.858143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.858031 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115"} err="failed to get container status \"daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115\": rpc error: code = NotFound desc = could not find container \"daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115\": container with ID starting with daa5cdfdcab721450774adb07b13b83773b4d280d6a8bb410f7d6ff98962a115 not found: ID does not exist" Oct 29 23:04:31.901592 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.901559 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955"] Oct 29 23:04:31.917776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:31.917745 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec4f6d-55d5d59f8c-q4955"] Oct 29 23:04:32.822448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:32.822412 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt"] Oct 29 23:04:32.833917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:32.833888 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-50734e-cdmxt"] Oct 29 23:04:33.068343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:33.068306 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78597dba-9df1-426d-b0bc-55665c8b3e33" path="/var/lib/kubelet/pods/78597dba-9df1-426d-b0bc-55665c8b3e33/volumes" Oct 29 23:04:33.068979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:33.068960 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d046e6-3059-4682-b985-b14ab9211fb8" path="/var/lib/kubelet/pods/d3d046e6-3059-4682-b985-b14ab9211fb8/volumes" Oct 29 23:04:33.069561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:33.069543 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5246614-6bd8-496d-b7e3-34a5ff531e5e" path="/var/lib/kubelet/pods/d5246614-6bd8-496d-b7e3-34a5ff531e5e/volumes" Oct 29 23:04:33.070560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:33.070533 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad" path="/var/lib/kubelet/pods/ee745440-fc8e-41a8-a7a6-b0dff4d6b1ad/volumes" Oct 29 23:04:39.027361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.027282 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp"] Oct 29 23:04:39.028096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.028076 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d5246614-6bd8-496d-b7e3-34a5ff531e5e" containerName="maas-api" Oct 29 23:04:39.028096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.028095 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5246614-6bd8-496d-b7e3-34a5ff531e5e" containerName="maas-api" Oct 29 23:04:39.028260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.028123 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d42eb596-4e68-4419-94d7-733d66c152c1" containerName="maas-api" Oct 29 23:04:39.028260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.028129 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42eb596-4e68-4419-94d7-733d66c152c1" containerName="maas-api" Oct 29 23:04:39.028260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.028235 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42eb596-4e68-4419-94d7-733d66c152c1" containerName="maas-api" Oct 29 23:04:39.028260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.028254 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5246614-6bd8-496d-b7e3-34a5ff531e5e" containerName="maas-api" Oct 29 23:04:39.033550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.033529 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.051836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.049846 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp"] Oct 29 23:04:39.071561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.071532 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:39.071561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.071612 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:04:39.083679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.083646 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.083881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.083859 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.083975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.083908 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj84q\" (UniqueName: \"kubernetes.io/projected/3d684390-a13d-442b-a6d0-be76c5eb369a-kube-api-access-wj84q\") pod \"maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.185988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.185950 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.186321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.186009 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj84q\" (UniqueName: \"kubernetes.io/projected/3d684390-a13d-442b-a6d0-be76c5eb369a-kube-api-access-wj84q\") pod \"maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.186321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.186208 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.186782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.186755 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.190368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.189233 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.209980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.209946 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj84q\" (UniqueName: \"kubernetes.io/projected/3d684390-a13d-442b-a6d0-be76c5eb369a-kube-api-access-wj84q\") pod \"maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.349715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.349625 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:39.665308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.665278 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp"] Oct 29 23:04:39.667918 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:04:39.667886 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d684390_a13d_442b_a6d0_be76c5eb369a.slice/crio-9eed57590d8e2669fc341da65f3da5dfc543a726b8a7b8293922079dcab6892c WatchSource:0}: Error finding container 9eed57590d8e2669fc341da65f3da5dfc543a726b8a7b8293922079dcab6892c: Status 404 returned error can't find the container with id 9eed57590d8e2669fc341da65f3da5dfc543a726b8a7b8293922079dcab6892c Oct 29 23:04:39.884334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.884298 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk"] Oct 29 23:04:39.884892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.884857 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" containerID="cri-o://b0b46be0e0490d1ff8beac818bd38e02e72bb3d56dc3c5a47294b941b0a7efae" gracePeriod=30 Oct 29 23:04:39.890528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.890495 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5"] Oct 29 23:04:39.890744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.890717 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" containerID="cri-o://fd85efcbc7902309c023f716a2fad29f17316a8504046fd5bb706695e25a2c23" gracePeriod=30 Oct 29 23:04:39.932416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:39.932337 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" event={"ID":"3d684390-a13d-442b-a6d0-be76c5eb369a","Type":"ContainerStarted","Data":"9eed57590d8e2669fc341da65f3da5dfc543a726b8a7b8293922079dcab6892c"} Oct 29 23:04:40.562771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.560572 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs"] Oct 29 23:04:40.567079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.567055 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.581381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.581347 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs"] Oct 29 23:04:40.651638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.651561 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjdq\" (UniqueName: \"kubernetes.io/projected/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-kube-api-access-jpjdq\") pod \"maas-api-revoke-keys-e2e-api-b-d430df-6rwvs\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.651801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.651738 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d430df-6rwvs\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.651801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.651790 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d430df-6rwvs\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.754190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.753924 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjdq\" (UniqueName: \"kubernetes.io/projected/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-kube-api-access-jpjdq\") pod \"maas-api-revoke-keys-e2e-api-b-d430df-6rwvs\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.754190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.754080 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d430df-6rwvs\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.754190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.754138 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d430df-6rwvs\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.755381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.755274 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d430df-6rwvs\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.757182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.757160 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d430df-6rwvs\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.778048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.778023 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjdq\" (UniqueName: \"kubernetes.io/projected/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-kube-api-access-jpjdq\") pod \"maas-api-revoke-keys-e2e-api-b-d430df-6rwvs\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:40.891653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:40.886198 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:41.107605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:41.107374 5736 generic.go:356] "Generic (PLEG): container finished" podID="3d684390-a13d-442b-a6d0-be76c5eb369a" containerID="d70da59f1724364147e5f4ab100db6db1e85af5d893e7314b69807aee51e7497" exitCode=0 Oct 29 23:04:41.107605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:41.107566 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" event={"ID":"3d684390-a13d-442b-a6d0-be76c5eb369a","Type":"ContainerDied","Data":"d70da59f1724364147e5f4ab100db6db1e85af5d893e7314b69807aee51e7497"} Oct 29 23:04:41.400719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:41.400689 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs"] Oct 29 23:04:41.402160 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:04:41.402132 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf1b4e9_e402_44b1_a49c_183de20d1c7d.slice/crio-54471a7f27de28d02ccfda7e3c81e3b7309170a3c82f77cf92af16949d1cdc30 WatchSource:0}: Error finding container 54471a7f27de28d02ccfda7e3c81e3b7309170a3c82f77cf92af16949d1cdc30: Status 404 returned error can't find the container with id 54471a7f27de28d02ccfda7e3c81e3b7309170a3c82f77cf92af16949d1cdc30 Oct 29 23:04:42.117212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.117102 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" event={"ID":"1bf1b4e9-e402-44b1-a49c-183de20d1c7d","Type":"ContainerStarted","Data":"eb157e76c8ed89233caff22255392cb3521cd132a3d274170da91dd14a5e7266"} Oct 29 23:04:42.117212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.117174 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" event={"ID":"1bf1b4e9-e402-44b1-a49c-183de20d1c7d","Type":"ContainerStarted","Data":"54471a7f27de28d02ccfda7e3c81e3b7309170a3c82f77cf92af16949d1cdc30"} Oct 29 23:04:42.159275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.159014 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" podStartSLOduration=2.15899713 podStartE2EDuration="2.15899713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:42.156835555 +0000 UTC m=+1279.272755161" watchObservedRunningTime="2026-10-29 23:04:42.15899713 +0000 UTC m=+1279.274916735" Oct 29 23:04:42.294096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.294071 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:42.403591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.403515 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d684390-a13d-442b-a6d0-be76c5eb369a-kube-api-access-wj84q\" (UniqueName: \"kubernetes.io/projected/3d684390-a13d-442b-a6d0-be76c5eb369a-kube-api-access-wj84q\") pod \"3d684390-a13d-442b-a6d0-be76c5eb369a\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " Oct 29 23:04:42.403731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.403698 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-openshift-ca\") pod \"3d684390-a13d-442b-a6d0-be76c5eb369a\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " Oct 29 23:04:42.403991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.403970 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-xks-ca\") pod \"3d684390-a13d-442b-a6d0-be76c5eb369a\" (UID: \"3d684390-a13d-442b-a6d0-be76c5eb369a\") " Oct 29 23:04:42.404093 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.404067 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-openshift-ca" pod "3d684390-a13d-442b-a6d0-be76c5eb369a" (UID: "3d684390-a13d-442b-a6d0-be76c5eb369a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:42.404624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.404575 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:42.405742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.405722 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d684390-a13d-442b-a6d0-be76c5eb369a-kube-api-access-wj84q" pod "3d684390-a13d-442b-a6d0-be76c5eb369a" (UID: "3d684390-a13d-442b-a6d0-be76c5eb369a"). InnerVolumeSpecName "kube-api-access-wj84q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:42.405914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.405898 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-xks-ca" pod "3d684390-a13d-442b-a6d0-be76c5eb369a" (UID: "3d684390-a13d-442b-a6d0-be76c5eb369a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:42.505967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.505939 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d684390-a13d-442b-a6d0-be76c5eb369a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:42.505967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:42.505968 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj84q\" (UniqueName: \"kubernetes.io/projected/3d684390-a13d-442b-a6d0-be76c5eb369a-kube-api-access-wj84q\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:43.146419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:43.140071 5736 generic.go:356] "Generic (PLEG): container finished" podID="1bf1b4e9-e402-44b1-a49c-183de20d1c7d" containerID="eb157e76c8ed89233caff22255392cb3521cd132a3d274170da91dd14a5e7266" exitCode=0 Oct 29 23:04:43.146419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:43.140192 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" event={"ID":"1bf1b4e9-e402-44b1-a49c-183de20d1c7d","Type":"ContainerDied","Data":"eb157e76c8ed89233caff22255392cb3521cd132a3d274170da91dd14a5e7266"} Oct 29 23:04:43.146419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:43.142876 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" event={"ID":"3d684390-a13d-442b-a6d0-be76c5eb369a","Type":"ContainerDied","Data":"9eed57590d8e2669fc341da65f3da5dfc543a726b8a7b8293922079dcab6892c"} Oct 29 23:04:43.146419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:43.142930 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eed57590d8e2669fc341da65f3da5dfc543a726b8a7b8293922079dcab6892c" Oct 29 23:04:43.146419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:43.142991 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp" Oct 29 23:04:44.022081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.022044 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:04:44.022249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.022109 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:04:44.029219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.029191 5736 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:04:44.029335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.029243 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:04:44.040683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.040653 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp"] Oct 29 23:04:44.051947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.051920 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82614eca-7bllp"] Oct 29 23:04:44.369473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.369448 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:44.454604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.448383 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-xks-ca\") pod \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " Oct 29 23:04:44.454604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.449162 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-openshift-ca\") pod \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " Oct 29 23:04:44.454604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.449332 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-kube-api-access-jpjdq\" (UniqueName: \"kubernetes.io/projected/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-kube-api-access-jpjdq\") pod \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\" (UID: \"1bf1b4e9-e402-44b1-a49c-183de20d1c7d\") " Oct 29 23:04:44.454604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.449520 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-openshift-ca" pod "1bf1b4e9-e402-44b1-a49c-183de20d1c7d" (UID: "1bf1b4e9-e402-44b1-a49c-183de20d1c7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:44.454604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.451205 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.454604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.453795 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-kube-api-access-jpjdq" pod "1bf1b4e9-e402-44b1-a49c-183de20d1c7d" (UID: "1bf1b4e9-e402-44b1-a49c-183de20d1c7d"). InnerVolumeSpecName "kube-api-access-jpjdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:44.455207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.454933 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-xks-ca" pod "1bf1b4e9-e402-44b1-a49c-183de20d1c7d" (UID: "1bf1b4e9-e402-44b1-a49c-183de20d1c7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:44.553638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.553596 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpjdq\" (UniqueName: \"kubernetes.io/projected/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-kube-api-access-jpjdq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:44.553638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:44.553640 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf1b4e9-e402-44b1-a49c-183de20d1c7d-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.014302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.013559 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94532a44-334e-48db-ad04-dd0ee1ac230f" path="/var/lib/kubelet/pods/94532a44-334e-48db-ad04-dd0ee1ac230f/volumes" Oct 29 23:04:45.181111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.177796 5736 generic.go:356] "Generic (PLEG): container finished" podID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerID="b0b46be0e0490d1ff8beac818bd38e02e72bb3d56dc3c5a47294b941b0a7efae" exitCode=0 Oct 29 23:04:45.181111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.177917 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" event={"ID":"f8f308b0-4938-435d-ac0d-9630aa87b92b","Type":"ContainerDied","Data":"b0b46be0e0490d1ff8beac818bd38e02e72bb3d56dc3c5a47294b941b0a7efae"} Oct 29 23:04:45.187194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.187158 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" event={"ID":"1bf1b4e9-e402-44b1-a49c-183de20d1c7d","Type":"ContainerDied","Data":"54471a7f27de28d02ccfda7e3c81e3b7309170a3c82f77cf92af16949d1cdc30"} Oct 29 23:04:45.187369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.187218 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54471a7f27de28d02ccfda7e3c81e3b7309170a3c82f77cf92af16949d1cdc30" Oct 29 23:04:45.187369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.187292 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs" Oct 29 23:04:45.195223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.193761 5736 generic.go:356] "Generic (PLEG): container finished" podID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerID="fd85efcbc7902309c023f716a2fad29f17316a8504046fd5bb706695e25a2c23" exitCode=0 Oct 29 23:04:45.195223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.193836 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" event={"ID":"8a2eb25d-0a9e-4028-9c70-c8baec507384","Type":"ContainerDied","Data":"fd85efcbc7902309c023f716a2fad29f17316a8504046fd5bb706695e25a2c23"} Oct 29 23:04:45.214880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.214853 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:04:45.258779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.258756 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:04:45.267871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.267846 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-kube-api-access-2vslj\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-kube-api-access-2vslj\") pod \"f8f308b0-4938-435d-ac0d-9630aa87b92b\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " Oct 29 23:04:45.268008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.267884 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-certs\") pod \"f8f308b0-4938-435d-ac0d-9630aa87b92b\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " Oct 29 23:04:45.268008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.267965 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-credential-socket\") pod \"f8f308b0-4938-435d-ac0d-9630aa87b92b\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " Oct 29 23:04:45.268125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.268005 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-podinfo\") pod \"f8f308b0-4938-435d-ac0d-9630aa87b92b\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " Oct 29 23:04:45.268125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.268055 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-envoy\") pod \"f8f308b0-4938-435d-ac0d-9630aa87b92b\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " Oct 29 23:04:45.268285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.268266 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-token\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-token\") pod \"f8f308b0-4938-435d-ac0d-9630aa87b92b\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " Oct 29 23:04:45.268377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.268332 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-credential-socket" pod "f8f308b0-4938-435d-ac0d-9630aa87b92b" (UID: "f8f308b0-4938-435d-ac0d-9630aa87b92b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.268438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.268413 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-data\") pod \"f8f308b0-4938-435d-ac0d-9630aa87b92b\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " Oct 29 23:04:45.268500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.268444 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f308b0-4938-435d-ac0d-9630aa87b92b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f308b0-4938-435d-ac0d-9630aa87b92b-istiod-ca-cert\") pod \"f8f308b0-4938-435d-ac0d-9630aa87b92b\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " Oct 29 23:04:45.268708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.268686 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-socket\") pod \"f8f308b0-4938-435d-ac0d-9630aa87b92b\" (UID: \"f8f308b0-4938-435d-ac0d-9630aa87b92b\") " Oct 29 23:04:45.268915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.268892 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-data" pod "f8f308b0-4938-435d-ac0d-9630aa87b92b" (UID: "f8f308b0-4938-435d-ac0d-9630aa87b92b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.269062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.269030 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f308b0-4938-435d-ac0d-9630aa87b92b-istiod-ca-cert" pod "f8f308b0-4938-435d-ac0d-9630aa87b92b" (UID: "f8f308b0-4938-435d-ac0d-9630aa87b92b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:45.269154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.269137 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-socket" pod "f8f308b0-4938-435d-ac0d-9630aa87b92b" (UID: "f8f308b0-4938-435d-ac0d-9630aa87b92b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.269565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.269543 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.269565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.269565 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.269765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.269574 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.269765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.269601 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f308b0-4938-435d-ac0d-9630aa87b92b-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.270072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.270049 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-certs" pod "f8f308b0-4938-435d-ac0d-9630aa87b92b" (UID: "f8f308b0-4938-435d-ac0d-9630aa87b92b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.271423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.271396 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-token" pod "f8f308b0-4938-435d-ac0d-9630aa87b92b" (UID: "f8f308b0-4938-435d-ac0d-9630aa87b92b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:45.271986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.271966 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-envoy" pod "f8f308b0-4938-435d-ac0d-9630aa87b92b" (UID: "f8f308b0-4938-435d-ac0d-9630aa87b92b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.272068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.271996 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-kube-api-access-2vslj" pod "f8f308b0-4938-435d-ac0d-9630aa87b92b" (UID: "f8f308b0-4938-435d-ac0d-9630aa87b92b"). InnerVolumeSpecName "kube-api-access-2vslj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:45.272183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.272161 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-podinfo" pod "f8f308b0-4938-435d-ac0d-9630aa87b92b" (UID: "f8f308b0-4938-435d-ac0d-9630aa87b92b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:45.370259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370231 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-token\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-token\") pod \"8a2eb25d-0a9e-4028-9c70-c8baec507384\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " Oct 29 23:04:45.370706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370341 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-certs\") pod \"8a2eb25d-0a9e-4028-9c70-c8baec507384\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " Oct 29 23:04:45.370706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370375 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-credential-socket\") pod \"8a2eb25d-0a9e-4028-9c70-c8baec507384\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " Oct 29 23:04:45.370706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370416 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-data\") pod \"8a2eb25d-0a9e-4028-9c70-c8baec507384\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " Oct 29 23:04:45.370706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370683 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-certs" pod "8a2eb25d-0a9e-4028-9c70-c8baec507384" (UID: "8a2eb25d-0a9e-4028-9c70-c8baec507384"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.370933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370725 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-data" pod "8a2eb25d-0a9e-4028-9c70-c8baec507384" (UID: "8a2eb25d-0a9e-4028-9c70-c8baec507384"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.370933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370751 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-podinfo\") pod \"8a2eb25d-0a9e-4028-9c70-c8baec507384\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " Oct 29 23:04:45.370933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370793 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-envoy\") pod \"8a2eb25d-0a9e-4028-9c70-c8baec507384\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " Oct 29 23:04:45.370933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370840 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-kube-api-access-fmp4w\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-kube-api-access-fmp4w\") pod \"8a2eb25d-0a9e-4028-9c70-c8baec507384\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " Oct 29 23:04:45.370933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370898 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-credential-socket" pod "8a2eb25d-0a9e-4028-9c70-c8baec507384" (UID: "8a2eb25d-0a9e-4028-9c70-c8baec507384"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.370933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370904 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2eb25d-0a9e-4028-9c70-c8baec507384-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2eb25d-0a9e-4028-9c70-c8baec507384-istiod-ca-cert\") pod \"8a2eb25d-0a9e-4028-9c70-c8baec507384\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " Oct 29 23:04:45.371249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.370975 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-socket\") pod \"8a2eb25d-0a9e-4028-9c70-c8baec507384\" (UID: \"8a2eb25d-0a9e-4028-9c70-c8baec507384\") " Oct 29 23:04:45.371249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.371223 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2eb25d-0a9e-4028-9c70-c8baec507384-istiod-ca-cert" pod "8a2eb25d-0a9e-4028-9c70-c8baec507384" (UID: "8a2eb25d-0a9e-4028-9c70-c8baec507384"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.371543 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-socket" pod "8a2eb25d-0a9e-4028-9c70-c8baec507384" (UID: "8a2eb25d-0a9e-4028-9c70-c8baec507384"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372573 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372612 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372626 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372640 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vslj\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-kube-api-access-2vslj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372654 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372668 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372681 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372693 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f308b0-4938-435d-ac0d-9630aa87b92b-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372704 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2eb25d-0a9e-4028-9c70-c8baec507384-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.372737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.372715 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.373984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.373961 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-token" pod "8a2eb25d-0a9e-4028-9c70-c8baec507384" (UID: "8a2eb25d-0a9e-4028-9c70-c8baec507384"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:45.374200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.374176 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-kube-api-access-fmp4w" pod "8a2eb25d-0a9e-4028-9c70-c8baec507384" (UID: "8a2eb25d-0a9e-4028-9c70-c8baec507384"). InnerVolumeSpecName "kube-api-access-fmp4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:45.374370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.374350 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-podinfo" pod "8a2eb25d-0a9e-4028-9c70-c8baec507384" (UID: "8a2eb25d-0a9e-4028-9c70-c8baec507384"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:45.375612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.375559 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-envoy" pod "8a2eb25d-0a9e-4028-9c70-c8baec507384" (UID: "8a2eb25d-0a9e-4028-9c70-c8baec507384"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:45.474298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.474264 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.474298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.474299 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.474729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.474313 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmp4w\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-kube-api-access-fmp4w\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:45.474729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:45.474330 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2eb25d-0a9e-4028-9c70-c8baec507384-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:46.205538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.205502 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" Oct 29 23:04:46.205538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.205531 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5" event={"ID":"8a2eb25d-0a9e-4028-9c70-c8baec507384","Type":"ContainerDied","Data":"548d5e29561557efa4febf5a1fdf0035b6d02b73f53577350c8356089bd4fc83"} Oct 29 23:04:46.205802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.205614 5736 scope.go:122] "RemoveContainer" containerID="fd85efcbc7902309c023f716a2fad29f17316a8504046fd5bb706695e25a2c23" Oct 29 23:04:46.207397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.207362 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" event={"ID":"f8f308b0-4938-435d-ac0d-9630aa87b92b","Type":"ContainerDied","Data":"a52d489f0f7951d2d67b449c7795ce1c375e281ffb33fe74dffd8d0a820404d1"} Oct 29 23:04:46.207544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.207404 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk" Oct 29 23:04:46.222051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.222033 5736 scope.go:122] "RemoveContainer" containerID="b0b46be0e0490d1ff8beac818bd38e02e72bb3d56dc3c5a47294b941b0a7efae" Oct 29 23:04:46.324603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.323883 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5"] Oct 29 23:04:46.334023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.331643 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-wdcv5"] Oct 29 23:04:46.339649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.339601 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk"] Oct 29 23:04:46.345365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:46.345325 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-7lcxk"] Oct 29 23:04:47.007198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:47.007160 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" path="/var/lib/kubelet/pods/8a2eb25d-0a9e-4028-9c70-c8baec507384/volumes" Oct 29 23:04:47.008024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:47.008003 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" path="/var/lib/kubelet/pods/f8f308b0-4938-435d-ac0d-9630aa87b92b/volumes" Oct 29 23:04:56.729473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:56.729427 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz"] Oct 29 23:04:56.729983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:56.729713 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" podUID="e5349e38-94a6-483b-a430-57bb302714d5" containerName="maas-api" containerID="cri-o://502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17" gracePeriod=30 Oct 29 23:04:56.829686 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:56.829647 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:04:57.060694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.060666 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:04:57.149537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.149488 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-metrics-certs\") pod \"e5349e38-94a6-483b-a430-57bb302714d5\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " Oct 29 23:04:57.149722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.149557 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5349e38-94a6-483b-a430-57bb302714d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5349e38-94a6-483b-a430-57bb302714d5-openshift-service-ca\") pod \"e5349e38-94a6-483b-a430-57bb302714d5\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " Oct 29 23:04:57.149722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.149708 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls\") pod \"e5349e38-94a6-483b-a430-57bb302714d5\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " Oct 29 23:04:57.149867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.149809 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5349e38-94a6-483b-a430-57bb302714d5-kube-api-access-hl2jz\" (UniqueName: \"kubernetes.io/projected/e5349e38-94a6-483b-a430-57bb302714d5-kube-api-access-hl2jz\") pod \"e5349e38-94a6-483b-a430-57bb302714d5\" (UID: \"e5349e38-94a6-483b-a430-57bb302714d5\") " Oct 29 23:04:57.151435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.151402 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5349e38-94a6-483b-a430-57bb302714d5-openshift-service-ca" pod "e5349e38-94a6-483b-a430-57bb302714d5" (UID: "e5349e38-94a6-483b-a430-57bb302714d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:57.155396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.154128 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-metrics-certs" pod "e5349e38-94a6-483b-a430-57bb302714d5" (UID: "e5349e38-94a6-483b-a430-57bb302714d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:57.161801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.155798 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls" pod "e5349e38-94a6-483b-a430-57bb302714d5" (UID: "e5349e38-94a6-483b-a430-57bb302714d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:57.162436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.162399 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5349e38-94a6-483b-a430-57bb302714d5-kube-api-access-hl2jz" pod "e5349e38-94a6-483b-a430-57bb302714d5" (UID: "e5349e38-94a6-483b-a430-57bb302714d5"). InnerVolumeSpecName "kube-api-access-hl2jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:57.251819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.251783 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:57.251819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.251815 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5349e38-94a6-483b-a430-57bb302714d5-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:57.251819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.251825 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5349e38-94a6-483b-a430-57bb302714d5-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:57.252029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.251836 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl2jz\" (UniqueName: \"kubernetes.io/projected/e5349e38-94a6-483b-a430-57bb302714d5-kube-api-access-hl2jz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:04:57.323651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.323554 5736 generic.go:356] "Generic (PLEG): container finished" podID="e5349e38-94a6-483b-a430-57bb302714d5" containerID="502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17" exitCode=0 Oct 29 23:04:57.323651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.323631 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" Oct 29 23:04:57.323850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.323646 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" event={"ID":"e5349e38-94a6-483b-a430-57bb302714d5","Type":"ContainerDied","Data":"502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17"} Oct 29 23:04:57.323850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.323708 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz" event={"ID":"e5349e38-94a6-483b-a430-57bb302714d5","Type":"ContainerDied","Data":"1cc9b09ac3b000b069b74286658a0fc12807d501fd63eba1d92f2afd9004de81"} Oct 29 23:04:57.323850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.323738 5736 scope.go:122] "RemoveContainer" containerID="502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17" Oct 29 23:04:57.340066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.340045 5736 scope.go:122] "RemoveContainer" containerID="502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17" Oct 29 23:04:57.340380 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:04:57.340361 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17\": container with ID starting with 502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17 not found: ID does not exist" containerID="502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17" Oct 29 23:04:57.340434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.340389 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17"} err="failed to get container status \"502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17\": rpc error: code = NotFound desc = could not find container \"502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17\": container with ID starting with 502078e4c93dfbb92ccd23a876fddec21b7d555feb606cc1d234e07c87cc1c17 not found: ID does not exist" Oct 29 23:04:57.406613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.394446 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz"] Oct 29 23:04:57.416867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:57.416833 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-10b990-f65f8cd8d-qc4dz"] Oct 29 23:04:58.290542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:58.290509 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp"] Oct 29 23:04:58.295890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:58.295855 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-10b990-7nglp"] Oct 29 23:04:59.005411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:59.005379 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d684390-a13d-442b-a6d0-be76c5eb369a" path="/var/lib/kubelet/pods/3d684390-a13d-442b-a6d0-be76c5eb369a/volumes" Oct 29 23:04:59.005875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:04:59.005861 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5349e38-94a6-483b-a430-57bb302714d5" path="/var/lib/kubelet/pods/e5349e38-94a6-483b-a430-57bb302714d5/volumes" Oct 29 23:05:02.027844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:02.027807 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq"] Oct 29 23:05:02.028261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:02.028012 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" containerID="cri-o://09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa" gracePeriod=30 Oct 29 23:05:04.775239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.775202 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk"] Oct 29 23:05:04.775910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.775876 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" Oct 29 23:05:04.775910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.775894 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" Oct 29 23:05:04.776019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.775912 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1bf1b4e9-e402-44b1-a49c-183de20d1c7d" containerName="revoke-keys" Oct 29 23:05:04.776019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.775918 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf1b4e9-e402-44b1-a49c-183de20d1c7d" containerName="revoke-keys" Oct 29 23:05:04.776019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.775936 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3d684390-a13d-442b-a6d0-be76c5eb369a" containerName="revoke-keys" Oct 29 23:05:04.776019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.775941 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d684390-a13d-442b-a6d0-be76c5eb369a" containerName="revoke-keys" Oct 29 23:05:04.776019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.776005 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e5349e38-94a6-483b-a430-57bb302714d5" containerName="maas-api" Oct 29 23:05:04.776019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.776014 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5349e38-94a6-483b-a430-57bb302714d5" containerName="maas-api" Oct 29 23:05:04.776235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.776036 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" Oct 29 23:05:04.776235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.776041 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" Oct 29 23:05:04.776302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.776273 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5349e38-94a6-483b-a430-57bb302714d5" containerName="maas-api" Oct 29 23:05:04.776343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.776305 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2eb25d-0a9e-4028-9c70-c8baec507384" containerName="istio-proxy" Oct 29 23:05:04.776343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.776322 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f308b0-4938-435d-ac0d-9630aa87b92b" containerName="istio-proxy" Oct 29 23:05:04.776343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.776331 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf1b4e9-e402-44b1-a49c-183de20d1c7d" containerName="revoke-keys" Oct 29 23:05:04.776441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.776344 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d684390-a13d-442b-a6d0-be76c5eb369a" containerName="revoke-keys" Oct 29 23:05:04.782293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.782269 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.798423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.798396 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b0c72c-openshift-default-dockercfg-twhkp\"" Oct 29 23:05:04.816433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.816391 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk"] Oct 29 23:05:04.842341 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.842308 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.842485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.842395 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddbbdc-958f-44f6-bf08-aad1877e8875-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.842546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.842519 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.842624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.842557 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.842624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.842606 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.842706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.842680 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.842979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.842959 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctqp\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-kube-api-access-9ctqp\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.843022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.843006 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.843064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.843050 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944200 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctqp\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-kube-api-access-9ctqp\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944248 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944285 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944333 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944386 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddbbdc-958f-44f6-bf08-aad1877e8875-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944446 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944471 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944499 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944573 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944738 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.944978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.944956 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.945253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.945225 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddbbdc-958f-44f6-bf08-aad1877e8875-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.945331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.945257 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.945477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.945449 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.947454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.947430 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.947573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.947551 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.967782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.967754 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctqp\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-kube-api-access-9ctqp\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:04.968109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:04.968085 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:05.102446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.098935 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:05.479686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.479640 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w"] Oct 29 23:05:05.481290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.480222 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" containerID="cri-o://adddc641767b19c24160b0f64d2b812c8e8673a077b3f20091e2cb813e345378" gracePeriod=30 Oct 29 23:05:05.496067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.496036 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk"] Oct 29 23:05:05.501186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.501159 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm"] Oct 29 23:05:05.501628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.501538 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" containerID="cri-o://12369565caf30742ac467486e0177197ffbc1a118a9c230731fd62748e990e18" gracePeriod=30 Oct 29 23:05:05.509092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.509065 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j"] Oct 29 23:05:05.509307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.509283 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" containerID="cri-o://a1f7d022077682f46939a91f38274004b7837495f56bceb79e18708d70d356ed" gracePeriod=30 Oct 29 23:05:05.517501 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:05:05.517466 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ddbbdc_958f_44f6_bf08_aad1877e8875.slice/crio-d42139e54723146b6436fb6f6bf844670f097110ea3a7a67d0c1bc36b81fb8b4 WatchSource:0}: Error finding container d42139e54723146b6436fb6f6bf844670f097110ea3a7a67d0c1bc36b81fb8b4: Status 404 returned error can't find the container with id d42139e54723146b6436fb6f6bf844670f097110ea3a7a67d0c1bc36b81fb8b4 Oct 29 23:05:05.520764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.520726 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:05:05.520845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.520812 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:05:05.520882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:05.520859 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:05:06.322603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:06.322109 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr"] Oct 29 23:05:06.333599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:06.326078 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c486c-kkcwr"] Oct 29 23:05:06.554724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:06.548824 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" event={"ID":"42ddbbdc-958f-44f6-bf08-aad1877e8875","Type":"ContainerStarted","Data":"db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92"} Oct 29 23:05:06.554724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:06.548954 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" event={"ID":"42ddbbdc-958f-44f6-bf08-aad1877e8875","Type":"ContainerStarted","Data":"d42139e54723146b6436fb6f6bf844670f097110ea3a7a67d0c1bc36b81fb8b4"} Oct 29 23:05:06.609296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:06.609176 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" podStartSLOduration=2.6091637949999997 podStartE2EDuration="2.609163795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:06.607201488 +0000 UTC m=+1303.723121093" watchObservedRunningTime="2026-10-29 23:05:06.609163795 +0000 UTC m=+1303.725083438" Oct 29 23:05:07.007145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.007110 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877b7cf3-5ac1-4e97-b2bc-b0a3129e2016" path="/var/lib/kubelet/pods/877b7cf3-5ac1-4e97-b2bc-b0a3129e2016/volumes" Oct 29 23:05:07.102559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.102533 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:07.104210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.104179 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:05:07.104334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.104228 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 23:05:07.479666 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.479632 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 23:05:07.592963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.590308 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-socket\") pod \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " Oct 29 23:05:07.592963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.590492 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-credential-socket\") pod \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " Oct 29 23:05:07.592963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.590538 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-podinfo\") pod \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " Oct 29 23:05:07.592963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.590610 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-envoy\") pod \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " Oct 29 23:05:07.592963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.590690 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-kube-api-access-m5n52\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-kube-api-access-m5n52\") pod \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " Oct 29 23:05:07.592963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.590908 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-data\") pod \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " Oct 29 23:05:07.592963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.590949 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-token\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-token\") pod \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " Oct 29 23:05:07.592963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.590984 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-certs\") pod \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " Oct 29 23:05:07.592963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.591011 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istiod-ca-cert\") pod \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\" (UID: \"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108\") " Oct 29 23:05:07.624603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.618734 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istiod-ca-cert" pod "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" (UID: "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:07.624603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.619074 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-data" pod "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" (UID: "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:07.624603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.620897 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-certs" pod "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" (UID: "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:07.624603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.621049 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-socket" pod "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" (UID: "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:07.624603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.621927 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-credential-socket" pod "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" (UID: "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:07.691613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.690360 5736 generic.go:356] "Generic (PLEG): container finished" podID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerID="09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa" exitCode=0 Oct 29 23:05:07.691613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.691508 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" Oct 29 23:05:07.696198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.692284 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" event={"ID":"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108","Type":"ContainerDied","Data":"09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa"} Oct 29 23:05:07.696198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.692341 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq" event={"ID":"8e48f4af-7aa3-41f1-9d6d-06aa2fee3108","Type":"ContainerDied","Data":"24756f629f8fec15d8a65ed160a1b625cdb600debed90da956bab03fd7a8a7fb"} Oct 29 23:05:07.696198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.692371 5736 scope.go:122] "RemoveContainer" containerID="09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa" Oct 29 23:05:07.721977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.720074 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.721977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.720113 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.721977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.720128 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.721977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.720142 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.721977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.720157 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.735856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.735814 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-token" pod "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" (UID: "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:07.736646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.736550 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-podinfo" pod "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" (UID: "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:07.737168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.737136 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-kube-api-access-m5n52" pod "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" (UID: "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108"). InnerVolumeSpecName "kube-api-access-m5n52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:07.743995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.743970 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-envoy" pod "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" (UID: "8e48f4af-7aa3-41f1-9d6d-06aa2fee3108"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:07.784681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.784535 5736 scope.go:122] "RemoveContainer" containerID="09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa" Oct 29 23:05:07.785022 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:05:07.784953 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa\": container with ID starting with 09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa not found: ID does not exist" containerID="09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa" Oct 29 23:05:07.785022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.784996 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa"} err="failed to get container status \"09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa\": rpc error: code = NotFound desc = could not find container \"09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa\": container with ID starting with 09fca0d09aef9a4b17e03664818f75551a5e1b0a350dbce9de1b4b19d7c356aa not found: ID does not exist" Oct 29 23:05:07.823635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.822210 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.823635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.822247 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.823635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.822263 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5n52\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-kube-api-access-m5n52\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:07.823635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:07.822278 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:08.098614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:08.097639 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq"] Oct 29 23:05:08.108601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:08.100498 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gj9hq"] Oct 29 23:05:08.116610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:08.111854 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:05:08.116610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:08.111924 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 23:05:09.007882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:09.007845 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" path="/var/lib/kubelet/pods/8e48f4af-7aa3-41f1-9d6d-06aa2fee3108/volumes" Oct 29 23:05:09.102782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:09.102747 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:05:09.102995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:09.102805 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 23:05:10.103552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:10.103514 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:05:10.104149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:10.103621 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 23:05:10.723552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:10.723297 5736 generic.go:356] "Generic (PLEG): container finished" podID="43c1dd77-c258-44c2-be49-ad19811704e9" containerID="12369565caf30742ac467486e0177197ffbc1a118a9c230731fd62748e990e18" exitCode=0 Oct 29 23:05:10.723552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:10.723445 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" event={"ID":"43c1dd77-c258-44c2-be49-ad19811704e9","Type":"ContainerDied","Data":"12369565caf30742ac467486e0177197ffbc1a118a9c230731fd62748e990e18"} Oct 29 23:05:10.725722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:10.725499 5736 generic.go:356] "Generic (PLEG): container finished" podID="452b7640-c7d1-4abf-9ade-695d97f29762" containerID="a1f7d022077682f46939a91f38274004b7837495f56bceb79e18708d70d356ed" exitCode=0 Oct 29 23:05:10.725722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:10.725663 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" event={"ID":"452b7640-c7d1-4abf-9ade-695d97f29762","Type":"ContainerDied","Data":"a1f7d022077682f46939a91f38274004b7837495f56bceb79e18708d70d356ed"} Oct 29 23:05:10.727623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:10.727566 5736 generic.go:356] "Generic (PLEG): container finished" podID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerID="adddc641767b19c24160b0f64d2b812c8e8673a077b3f20091e2cb813e345378" exitCode=0 Oct 29 23:05:10.727752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:10.727697 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" event={"ID":"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81","Type":"ContainerDied","Data":"adddc641767b19c24160b0f64d2b812c8e8673a077b3f20091e2cb813e345378"} Oct 29 23:05:11.162860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.131550 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 23:05:11.162860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.131636 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 23:05:11.292414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.292380 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:05:11.383242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.383217 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-kube-api-access-v649c\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-kube-api-access-v649c\") pod \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " Oct 29 23:05:11.383379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.383314 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-credential-socket\") pod \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " Oct 29 23:05:11.383502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.383484 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-envoy\") pod \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " Oct 29 23:05:11.383630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.383573 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-podinfo\") pod \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " Oct 29 23:05:11.383754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.383739 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-socket\") pod \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " Oct 29 23:05:11.383889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.383875 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istiod-ca-cert\") pod \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " Oct 29 23:05:11.383976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.383963 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-certs\") pod \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " Oct 29 23:05:11.384038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.384022 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-data\") pod \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " Oct 29 23:05:11.384096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.384066 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-token\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-token\") pod \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\" (UID: \"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81\") " Oct 29 23:05:11.384549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.384523 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istiod-ca-cert" pod "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" (UID: "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:11.384769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.384739 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.384882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.384737 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:05:11.384951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.384892 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-credential-socket" pod "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" (UID: "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.389109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.389073 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-envoy" pod "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" (UID: "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.389661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.389568 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-socket" pod "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" (UID: "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.389759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.389669 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-kube-api-access-v649c" pod "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" (UID: "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81"). InnerVolumeSpecName "kube-api-access-v649c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:11.389879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.389835 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-data" pod "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" (UID: "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.390132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.390105 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-certs" pod "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" (UID: "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.390710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.390678 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-podinfo" pod "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" (UID: "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:11.393452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.393429 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-token" pod "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" (UID: "fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:11.395234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.395178 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486028 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-data\") pod \"452b7640-c7d1-4abf-9ade-695d97f29762\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486080 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/452b7640-c7d1-4abf-9ade-695d97f29762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452b7640-c7d1-4abf-9ade-695d97f29762-istio-podinfo\") pod \"452b7640-c7d1-4abf-9ade-695d97f29762\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486117 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-data\") pod \"43c1dd77-c258-44c2-be49-ad19811704e9\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486165 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-envoy\") pod \"43c1dd77-c258-44c2-be49-ad19811704e9\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486367 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/452b7640-c7d1-4abf-9ade-695d97f29762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452b7640-c7d1-4abf-9ade-695d97f29762-istiod-ca-cert\") pod \"452b7640-c7d1-4abf-9ade-695d97f29762\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486412 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-certs\") pod \"43c1dd77-c258-44c2-be49-ad19811704e9\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486441 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-kube-api-access-rwwr4\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-kube-api-access-rwwr4\") pod \"452b7640-c7d1-4abf-9ade-695d97f29762\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486503 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-envoy\") pod \"452b7640-c7d1-4abf-9ade-695d97f29762\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486551 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-certs\") pod \"452b7640-c7d1-4abf-9ade-695d97f29762\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486623 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-credential-socket\") pod \"43c1dd77-c258-44c2-be49-ad19811704e9\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486718 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-credential-socket\") pod \"452b7640-c7d1-4abf-9ade-695d97f29762\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486751 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-socket\") pod \"452b7640-c7d1-4abf-9ade-695d97f29762\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " Oct 29 23:05:11.486793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486785 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c1dd77-c258-44c2-be49-ad19811704e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c1dd77-c258-44c2-be49-ad19811704e9-istiod-ca-cert\") pod \"43c1dd77-c258-44c2-be49-ad19811704e9\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " Oct 29 23:05:11.487594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486815 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-istio-token\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-istio-token\") pod \"452b7640-c7d1-4abf-9ade-695d97f29762\" (UID: \"452b7640-c7d1-4abf-9ade-695d97f29762\") " Oct 29 23:05:11.487594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486861 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-socket\") pod \"43c1dd77-c258-44c2-be49-ad19811704e9\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " Oct 29 23:05:11.487594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486889 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43c1dd77-c258-44c2-be49-ad19811704e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c1dd77-c258-44c2-be49-ad19811704e9-istio-podinfo\") pod \"43c1dd77-c258-44c2-be49-ad19811704e9\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " Oct 29 23:05:11.487594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486930 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-kube-api-access-zvl6s\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-kube-api-access-zvl6s\") pod \"43c1dd77-c258-44c2-be49-ad19811704e9\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " Oct 29 23:05:11.487594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.486962 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-istio-token\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-istio-token\") pod \"43c1dd77-c258-44c2-be49-ad19811704e9\" (UID: \"43c1dd77-c258-44c2-be49-ad19811704e9\") " Oct 29 23:05:11.489970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.489943 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.490157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.490143 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.490271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.490260 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.490381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.490367 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v649c\" (UniqueName: \"kubernetes.io/projected/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-kube-api-access-v649c\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.490483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.490470 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.490572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.490560 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.490693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.490680 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.490783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.490770 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.493725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.493701 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-certs" pod "452b7640-c7d1-4abf-9ade-695d97f29762" (UID: "452b7640-c7d1-4abf-9ade-695d97f29762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.494053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.494036 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-data" pod "452b7640-c7d1-4abf-9ade-695d97f29762" (UID: "452b7640-c7d1-4abf-9ade-695d97f29762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.497020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.496766 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/452b7640-c7d1-4abf-9ade-695d97f29762-istio-podinfo" pod "452b7640-c7d1-4abf-9ade-695d97f29762" (UID: "452b7640-c7d1-4abf-9ade-695d97f29762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:11.498005 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.497982 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-certs" pod "43c1dd77-c258-44c2-be49-ad19811704e9" (UID: "43c1dd77-c258-44c2-be49-ad19811704e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.498352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.498152 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-data" pod "43c1dd77-c258-44c2-be49-ad19811704e9" (UID: "43c1dd77-c258-44c2-be49-ad19811704e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.498596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.498488 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-kube-api-access-rwwr4" pod "452b7640-c7d1-4abf-9ade-695d97f29762" (UID: "452b7640-c7d1-4abf-9ade-695d97f29762"). InnerVolumeSpecName "kube-api-access-rwwr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:11.498967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.498823 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-credential-socket" pod "452b7640-c7d1-4abf-9ade-695d97f29762" (UID: "452b7640-c7d1-4abf-9ade-695d97f29762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.502361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.499102 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-credential-socket" pod "43c1dd77-c258-44c2-be49-ad19811704e9" (UID: "43c1dd77-c258-44c2-be49-ad19811704e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.503195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.502927 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-envoy" pod "452b7640-c7d1-4abf-9ade-695d97f29762" (UID: "452b7640-c7d1-4abf-9ade-695d97f29762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.506209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.506033 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-istio-token" pod "452b7640-c7d1-4abf-9ade-695d97f29762" (UID: "452b7640-c7d1-4abf-9ade-695d97f29762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:11.506601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.506345 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-socket" pod "452b7640-c7d1-4abf-9ade-695d97f29762" (UID: "452b7640-c7d1-4abf-9ade-695d97f29762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.515873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.509866 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452b7640-c7d1-4abf-9ade-695d97f29762-istiod-ca-cert" pod "452b7640-c7d1-4abf-9ade-695d97f29762" (UID: "452b7640-c7d1-4abf-9ade-695d97f29762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:11.515873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.513490 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-socket" pod "43c1dd77-c258-44c2-be49-ad19811704e9" (UID: "43c1dd77-c258-44c2-be49-ad19811704e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.515873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.513685 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c1dd77-c258-44c2-be49-ad19811704e9-istiod-ca-cert" pod "43c1dd77-c258-44c2-be49-ad19811704e9" (UID: "43c1dd77-c258-44c2-be49-ad19811704e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:11.520894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.520848 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-kube-api-access-zvl6s" pod "43c1dd77-c258-44c2-be49-ad19811704e9" (UID: "43c1dd77-c258-44c2-be49-ad19811704e9"). InnerVolumeSpecName "kube-api-access-zvl6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:11.521515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.521475 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43c1dd77-c258-44c2-be49-ad19811704e9-istio-podinfo" pod "43c1dd77-c258-44c2-be49-ad19811704e9" (UID: "43c1dd77-c258-44c2-be49-ad19811704e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:11.523149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.523117 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-istio-token" pod "43c1dd77-c258-44c2-be49-ad19811704e9" (UID: "43c1dd77-c258-44c2-be49-ad19811704e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:11.523663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.523629 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-envoy" pod "43c1dd77-c258-44c2-be49-ad19811704e9" (UID: "43c1dd77-c258-44c2-be49-ad19811704e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:11.592246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592221 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592248 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c1dd77-c258-44c2-be49-ad19811704e9-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592258 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvl6s\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-kube-api-access-zvl6s\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592267 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c1dd77-c258-44c2-be49-ad19811704e9-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592276 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592283 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452b7640-c7d1-4abf-9ade-695d97f29762-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592291 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592299 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592307 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452b7640-c7d1-4abf-9ade-695d97f29762-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592315 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592326 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwwr4\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-kube-api-access-rwwr4\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592333 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592343 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592350 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c1dd77-c258-44c2-be49-ad19811704e9-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592360 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592368 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452b7640-c7d1-4abf-9ade-695d97f29762-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592379 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c1dd77-c258-44c2-be49-ad19811704e9-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.592384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.592387 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452b7640-c7d1-4abf-9ade-695d97f29762-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.761254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.761167 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" Oct 29 23:05:11.761254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.761190 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w" event={"ID":"fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81","Type":"ContainerDied","Data":"0f53ca92487abfcaedac69470e91e53acd30e2dfa89077e2a462e0b9e9b665a7"} Oct 29 23:05:11.761254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.761252 5736 scope.go:122] "RemoveContainer" containerID="adddc641767b19c24160b0f64d2b812c8e8673a077b3f20091e2cb813e345378" Oct 29 23:05:11.762875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.762855 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" Oct 29 23:05:11.762949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.762885 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm" event={"ID":"43c1dd77-c258-44c2-be49-ad19811704e9","Type":"ContainerDied","Data":"fb53ff32beee70fd87af65bf343aef08f0057e31909966fdfc81e49f3521f092"} Oct 29 23:05:11.767851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.767829 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" Oct 29 23:05:11.767988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.767880 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j" event={"ID":"452b7640-c7d1-4abf-9ade-695d97f29762","Type":"ContainerDied","Data":"fb3f92fc1c43eccc2d3c831a45ebe0c9ba6f026940b29ce26ad7da72f9b56567"} Oct 29 23:05:11.782551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.779622 5736 scope.go:122] "RemoveContainer" containerID="12369565caf30742ac467486e0177197ffbc1a118a9c230731fd62748e990e18" Oct 29 23:05:11.791948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.791925 5736 scope.go:122] "RemoveContainer" containerID="a1f7d022077682f46939a91f38274004b7837495f56bceb79e18708d70d356ed" Oct 29 23:05:11.855670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.841265 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm"] Oct 29 23:05:11.857893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.857848 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-cjkrm"] Oct 29 23:05:11.890241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.890212 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j"] Oct 29 23:05:11.894497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.894467 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-vhm2j"] Oct 29 23:05:11.928418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.928392 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w"] Oct 29 23:05:11.941710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:11.941434 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-rsb6w"] Oct 29 23:05:12.196386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:12.196291 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:12.197247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:12.197186 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:12.198442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:12.198418 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:05:13.012727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:13.012687 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" path="/var/lib/kubelet/pods/43c1dd77-c258-44c2-be49-ad19811704e9/volumes" Oct 29 23:05:13.013569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:13.013546 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" path="/var/lib/kubelet/pods/452b7640-c7d1-4abf-9ade-695d97f29762/volumes" Oct 29 23:05:13.014397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:13.014377 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" path="/var/lib/kubelet/pods/fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81/volumes" Oct 29 23:05:23.943422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:23.943388 5736 scope.go:122] "RemoveContainer" containerID="e74575280fa2b07691c1301c205bb97b5f451fd1a69c5a981c131a80a6a124ed" Oct 29 23:05:23.955228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:23.955201 5736 scope.go:122] "RemoveContainer" containerID="05c9596e065ad00d3bf1e20310170b9ab07df844cf2a07ef15ceb2733759f8d1" Oct 29 23:05:23.965096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:23.965065 5736 scope.go:122] "RemoveContainer" containerID="f1abc4325abbdc64447622392e4ef84e9c479b0b2b2f3337afd1fd668156654e" Oct 29 23:05:23.974149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:23.974126 5736 scope.go:122] "RemoveContainer" containerID="24ff1b883efd4a24c9268be34423275771e44cd3a5f0271b9a4d71717fdce8f1" Oct 29 23:05:23.983385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:23.983354 5736 scope.go:122] "RemoveContainer" containerID="4d61f53add7de6ce26b2b9efe059c5e8e8f8891851d62761289258acbf3f2981" Oct 29 23:05:26.823732 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:05:26.823699 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:05:49.904904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.904869 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth"] Oct 29 23:05:49.912927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.912892 5736 cpu_manager.go:414] "containerMap: removing container" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" Oct 29 23:05:49.912927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.912920 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" Oct 29 23:05:49.913158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.912995 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" Oct 29 23:05:49.913158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.913004 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" Oct 29 23:05:49.913158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.913017 5736 cpu_manager.go:414] "containerMap: removing container" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" Oct 29 23:05:49.913158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.913025 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" Oct 29 23:05:49.913158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.913045 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" Oct 29 23:05:49.913158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.913052 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" Oct 29 23:05:49.913465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.913192 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e48f4af-7aa3-41f1-9d6d-06aa2fee3108" containerName="istio-proxy" Oct 29 23:05:49.913465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.913206 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="452b7640-c7d1-4abf-9ade-695d97f29762" containerName="istio-proxy" Oct 29 23:05:49.913465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.913217 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcec88bf-6f9f-4d99-bd0b-0ec5c6dd9a81" containerName="istio-proxy" Oct 29 23:05:49.913465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.913230 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c1dd77-c258-44c2-be49-ad19811704e9" containerName="istio-proxy" Oct 29 23:05:49.917236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.917211 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:49.929991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:49.929966 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth"] Oct 29 23:05:50.120341 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.106372 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024e7072-b667-4a09-ab8e-6eadbce1c464-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.134937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.134903 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.135347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.135316 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.141796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.135532 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.142258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.142229 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.151717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.142556 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqnqf\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-kube-api-access-jqnqf\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.152789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.152762 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.153055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.153037 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.165555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.165392 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.278109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.278062 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024e7072-b667-4a09-ab8e-6eadbce1c464-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.296188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.295751 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.296188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.295849 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.296188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.295891 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.296188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.295996 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.296188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.296106 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqnqf\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-kube-api-access-jqnqf\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.296188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.296142 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.296649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.296212 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.296649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.296244 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.296916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.296888 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.297479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.297453 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.297726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.297709 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.301052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.292476 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024e7072-b667-4a09-ab8e-6eadbce1c464-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.302055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.302022 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.306198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.306170 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.322276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.322115 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.331189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.331158 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.332041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.332015 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqnqf\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-kube-api-access-jqnqf\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.541652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.541609 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:50.869161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.869135 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth"] Oct 29 23:05:50.871275 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:05:50.871231 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024e7072_b667_4a09_ab8e_6eadbce1c464.slice/crio-b8d40ba52a24c99d4b2e2e17ea757367625f68e94e37ecf2803583e2b80be714 WatchSource:0}: Error finding container b8d40ba52a24c99d4b2e2e17ea757367625f68e94e37ecf2803583e2b80be714: Status 404 returned error can't find the container with id b8d40ba52a24c99d4b2e2e17ea757367625f68e94e37ecf2803583e2b80be714 Oct 29 23:05:50.874167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.874135 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:05:50.874271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.874201 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:05:50.874271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:50.874230 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:05:51.133863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:51.133761 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs"] Oct 29 23:05:51.137278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:51.137251 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d430df-6rwvs"] Oct 29 23:05:51.401134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:51.401031 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" event={"ID":"024e7072-b667-4a09-ab8e-6eadbce1c464","Type":"ContainerStarted","Data":"608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c"} Oct 29 23:05:51.401134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:51.401093 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" event={"ID":"024e7072-b667-4a09-ab8e-6eadbce1c464","Type":"ContainerStarted","Data":"b8d40ba52a24c99d4b2e2e17ea757367625f68e94e37ecf2803583e2b80be714"} Oct 29 23:05:51.459881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:51.459816 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" podStartSLOduration=2.459799968 podStartE2EDuration="2.459799968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:51.456064168 +0000 UTC m=+1348.571983776" watchObservedRunningTime="2026-10-29 23:05:51.459799968 +0000 UTC m=+1348.575719642" Oct 29 23:05:51.554604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:51.553923 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:51.558840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:51.558805 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 23:05:51.559021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:51.558860 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 23:05:52.554955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:52.554919 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 23:05:52.555551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:52.555519 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 23:05:53.017681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:53.014890 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf1b4e9-e402-44b1-a49c-183de20d1c7d" path="/var/lib/kubelet/pods/1bf1b4e9-e402-44b1-a49c-183de20d1c7d/volumes" Oct 29 23:05:53.550604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:53.548015 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 23:05:53.550604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:53.548079 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 23:05:54.545031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:54.544992 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 23:05:54.555803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:54.555759 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 23:05:55.551968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:55.551927 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 23:05:55.553855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:55.553823 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 23:05:56.556066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:56.556021 5736 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 23:05:56.556531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:56.556091 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 23:05:56.826635 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:05:56.826515 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:05:57.546680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:57.546647 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:57.547047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:57.547022 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:57.547711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:57.547691 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:05:58.490203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:58.490167 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w"] Oct 29 23:05:58.491056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:58.491004 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" containerID="cri-o://4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb" gracePeriod=30 Oct 29 23:05:59.943524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:59.943486 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x"] Oct 29 23:05:59.948818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:59.948794 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:05:59.970885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:59.970852 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-921680-openshift-default-dockercfg-6zskc\"" Oct 29 23:05:59.984153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:05:59.984117 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x"] Oct 29 23:06:00.100772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.072462 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-envoy\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.141391 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.141356 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-data\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.163515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.163480 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-credential-socket\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.163884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.163862 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-socket\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.164048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.164032 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-certs\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.164468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.164446 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-istio-token\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.164715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.164695 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79fd910-a277-4b22-849e-bead58762990-istio-podinfo\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.165020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.165004 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79fd910-a277-4b22-849e-bead58762990-istiod-ca-cert\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.165221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.165202 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkq7\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-kube-api-access-4bkq7\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.269171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.269115 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-istio-token\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.269390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.269194 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79fd910-a277-4b22-849e-bead58762990-istio-podinfo\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.269467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.269384 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79fd910-a277-4b22-849e-bead58762990-istiod-ca-cert\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.269467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.269436 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkq7\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-kube-api-access-4bkq7\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.269638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.269466 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-envoy\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.269638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.269499 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-data\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.269638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.269565 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-credential-socket\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.269974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.269951 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-credential-socket\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.270107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.270089 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-socket\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.270171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.270123 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-certs\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.270637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.270566 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-certs\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.272204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.272156 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79fd910-a277-4b22-849e-bead58762990-istiod-ca-cert\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.272505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.272433 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-socket\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.272685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.272596 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-data\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.283669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.277357 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79fd910-a277-4b22-849e-bead58762990-istio-podinfo\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.298505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.298459 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-envoy\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.312517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.310543 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkq7\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-kube-api-access-4bkq7\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.317487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.317353 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-istio-token\") pod \"e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.578438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.578347 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:00.913235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.912811 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x"] Oct 29 23:06:00.923138 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:06:00.923092 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79fd910_a277_4b22_849e_bead58762990.slice/crio-470dbd9da645e03c3d10e48b1907f83ff6808f9190619ffa7e5401c6b55b1be2 WatchSource:0}: Error finding container 470dbd9da645e03c3d10e48b1907f83ff6808f9190619ffa7e5401c6b55b1be2: Status 404 returned error can't find the container with id 470dbd9da645e03c3d10e48b1907f83ff6808f9190619ffa7e5401c6b55b1be2 Oct 29 23:06:00.926108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.926071 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:06:00.926215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.926141 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:06:00.926215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:00.926170 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:06:01.520329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:01.520277 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" event={"ID":"f79fd910-a277-4b22-849e-bead58762990","Type":"ContainerStarted","Data":"cb80dd9eb8bec558e57f778e4f30a14b216e33c397a2f790ed61b84f3e297f24"} Oct 29 23:06:01.520865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:01.520401 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" event={"ID":"f79fd910-a277-4b22-849e-bead58762990","Type":"ContainerStarted","Data":"470dbd9da645e03c3d10e48b1907f83ff6808f9190619ffa7e5401c6b55b1be2"} Oct 29 23:06:01.578639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:01.578604 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:01.585016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:01.584978 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:01.613401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:01.613341 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" podStartSLOduration=2.613327434 podStartE2EDuration="2.613327434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:01.612541512 +0000 UTC m=+1358.728461118" watchObservedRunningTime="2026-10-29 23:06:01.613327434 +0000 UTC m=+1358.729247039" Oct 29 23:06:02.527720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:02.527682 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:02.528718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:02.528697 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:06:03.804368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.804337 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:06:03.864705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.864612 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-data\") pod \"e8984147-87bd-4105-989b-e2beb829eef4\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " Oct 29 23:06:03.864705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.864678 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-credential-socket\") pod \"e8984147-87bd-4105-989b-e2beb829eef4\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " Oct 29 23:06:03.864930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.864708 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-socket\") pod \"e8984147-87bd-4105-989b-e2beb829eef4\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " Oct 29 23:06:03.864930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.864744 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8984147-87bd-4105-989b-e2beb829eef4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8984147-87bd-4105-989b-e2beb829eef4-istio-podinfo\") pod \"e8984147-87bd-4105-989b-e2beb829eef4\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " Oct 29 23:06:03.864930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.864855 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-kube-api-access-tcdqq\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-kube-api-access-tcdqq\") pod \"e8984147-87bd-4105-989b-e2beb829eef4\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " Oct 29 23:06:03.865097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.864936 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8984147-87bd-4105-989b-e2beb829eef4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8984147-87bd-4105-989b-e2beb829eef4-istiod-ca-cert\") pod \"e8984147-87bd-4105-989b-e2beb829eef4\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " Oct 29 23:06:03.865097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.864999 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-certs\") pod \"e8984147-87bd-4105-989b-e2beb829eef4\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " Oct 29 23:06:03.865192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.865141 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-envoy\") pod \"e8984147-87bd-4105-989b-e2beb829eef4\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " Oct 29 23:06:03.865251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.865184 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-socket" pod "e8984147-87bd-4105-989b-e2beb829eef4" (UID: "e8984147-87bd-4105-989b-e2beb829eef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:03.865251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.865201 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-credential-socket" pod "e8984147-87bd-4105-989b-e2beb829eef4" (UID: "e8984147-87bd-4105-989b-e2beb829eef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:03.865364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.865343 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-istio-token\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-istio-token\") pod \"e8984147-87bd-4105-989b-e2beb829eef4\" (UID: \"e8984147-87bd-4105-989b-e2beb829eef4\") " Oct 29 23:06:03.865442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.865415 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-certs" pod "e8984147-87bd-4105-989b-e2beb829eef4" (UID: "e8984147-87bd-4105-989b-e2beb829eef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:03.865505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.865450 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8984147-87bd-4105-989b-e2beb829eef4-istiod-ca-cert" pod "e8984147-87bd-4105-989b-e2beb829eef4" (UID: "e8984147-87bd-4105-989b-e2beb829eef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:03.865875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.865840 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-data" pod "e8984147-87bd-4105-989b-e2beb829eef4" (UID: "e8984147-87bd-4105-989b-e2beb829eef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:03.866172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.866131 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:03.866172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.866163 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:03.866343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.866178 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:03.866343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.866193 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8984147-87bd-4105-989b-e2beb829eef4-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:03.866343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.866206 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:03.868194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.868167 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8984147-87bd-4105-989b-e2beb829eef4-istio-podinfo" pod "e8984147-87bd-4105-989b-e2beb829eef4" (UID: "e8984147-87bd-4105-989b-e2beb829eef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:03.868194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.868180 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-istio-token" pod "e8984147-87bd-4105-989b-e2beb829eef4" (UID: "e8984147-87bd-4105-989b-e2beb829eef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:03.868867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.868835 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-kube-api-access-tcdqq" pod "e8984147-87bd-4105-989b-e2beb829eef4" (UID: "e8984147-87bd-4105-989b-e2beb829eef4"). InnerVolumeSpecName "kube-api-access-tcdqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:03.869111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.869085 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-envoy" pod "e8984147-87bd-4105-989b-e2beb829eef4" (UID: "e8984147-87bd-4105-989b-e2beb829eef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:03.968157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.968123 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:03.968157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.968154 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8984147-87bd-4105-989b-e2beb829eef4-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:03.968157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.968164 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcdqq\" (UniqueName: \"kubernetes.io/projected/e8984147-87bd-4105-989b-e2beb829eef4-kube-api-access-tcdqq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:03.968400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:03.968176 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8984147-87bd-4105-989b-e2beb829eef4-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:04.544740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.544698 5736 generic.go:356] "Generic (PLEG): container finished" podID="e8984147-87bd-4105-989b-e2beb829eef4" containerID="4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb" exitCode=0 Oct 29 23:06:04.544941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.544783 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" Oct 29 23:06:04.544941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.544782 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" event={"ID":"e8984147-87bd-4105-989b-e2beb829eef4","Type":"ContainerDied","Data":"4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb"} Oct 29 23:06:04.544941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.544880 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w" event={"ID":"e8984147-87bd-4105-989b-e2beb829eef4","Type":"ContainerDied","Data":"03e75976ce70192c231f17d7d701c6267218162b91f22baccec0599481faee99"} Oct 29 23:06:04.544941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.544911 5736 scope.go:122] "RemoveContainer" containerID="4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb" Oct 29 23:06:04.557480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.557457 5736 scope.go:122] "RemoveContainer" containerID="eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7" Oct 29 23:06:04.573107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.573086 5736 scope.go:122] "RemoveContainer" containerID="4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb" Oct 29 23:06:04.573507 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:06:04.573484 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb\": container with ID starting with 4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb not found: ID does not exist" containerID="4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb" Oct 29 23:06:04.573615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.573515 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb"} err="failed to get container status \"4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb\": rpc error: code = NotFound desc = could not find container \"4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb\": container with ID starting with 4022d35127c0f7e9a76876da12d7a3e0b2e6efac125c05373600cea2f2a90cfb not found: ID does not exist" Oct 29 23:06:04.573615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.573539 5736 scope.go:122] "RemoveContainer" containerID="eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7" Oct 29 23:06:04.574093 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:06:04.574066 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7\": container with ID starting with eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7 not found: ID does not exist" containerID="eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7" Oct 29 23:06:04.574171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.574101 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7"} err="failed to get container status \"eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7\": rpc error: code = NotFound desc = could not find container \"eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7\": container with ID starting with eb14d0bdbf62d12f6ba9e0720876a6d4012db4db67f08d88baf4d421e94dd8f7 not found: ID does not exist" Oct 29 23:06:04.610956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.610924 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w"] Oct 29 23:06:04.622711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:04.622682 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d430df-openshift-default-69889fb89c-5tn8w"] Oct 29 23:06:05.005077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.005035 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8984147-87bd-4105-989b-e2beb829eef4" path="/var/lib/kubelet/pods/e8984147-87bd-4105-989b-e2beb829eef4/volumes" Oct 29 23:06:05.842303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.840753 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262"] Oct 29 23:06:05.842303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.841714 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" Oct 29 23:06:05.842303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.841732 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" Oct 29 23:06:05.842303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.841783 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" Oct 29 23:06:05.842303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.841791 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" Oct 29 23:06:05.842303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.841963 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" Oct 29 23:06:05.842303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.841983 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8984147-87bd-4105-989b-e2beb829eef4" containerName="istio-proxy" Oct 29 23:06:05.865608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.854800 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262"] Oct 29 23:06:05.865608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.854929 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:05.887594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.878421 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-957eef-openshift-default-dockercfg-fccr8\"" Oct 29 23:06:06.019605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:05.995349 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-credential-socket\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.047820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.040139 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k59dz\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-kube-api-access-k59dz\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.047820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.040217 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-istio-token\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.047820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.040259 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9569f274-5c9b-4a99-81f2-7321183979b2-istio-podinfo\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.047820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.040401 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-envoy\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.047820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.040554 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-data\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.047820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.040677 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-socket\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.047820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.041713 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-certs\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.047820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.041832 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9569f274-5c9b-4a99-81f2-7321183979b2-istiod-ca-cert\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.144155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.144052 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k59dz\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-kube-api-access-k59dz\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.144468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.144436 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-istio-token\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.144696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.144675 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9569f274-5c9b-4a99-81f2-7321183979b2-istio-podinfo\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.144979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.144958 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-envoy\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.145149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.145116 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-data\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.145251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.145228 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-socket\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.145374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.145279 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-certs\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.145374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.145368 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9569f274-5c9b-4a99-81f2-7321183979b2-istiod-ca-cert\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.145507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.145431 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-credential-socket\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.146349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.146320 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-certs\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.146497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.146476 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9569f274-5c9b-4a99-81f2-7321183979b2-istiod-ca-cert\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.146612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.146572 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-data\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.146764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.146741 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-credential-socket\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.146950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.146932 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-socket\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.152616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.150359 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9569f274-5c9b-4a99-81f2-7321183979b2-istio-podinfo\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.152616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.151218 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-envoy\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.168403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.168370 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-istio-token\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.168560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.168465 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k59dz\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-kube-api-access-k59dz\") pod \"e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.251849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.251812 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:06.657853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.657810 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:06:06.658304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.658193 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:06:06.658494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.658463 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:06:06.660032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:06.660003 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262"] Oct 29 23:06:07.654049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:07.654000 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" event={"ID":"9569f274-5c9b-4a99-81f2-7321183979b2","Type":"ContainerStarted","Data":"045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb"} Oct 29 23:06:07.654510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:07.654117 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" event={"ID":"9569f274-5c9b-4a99-81f2-7321183979b2","Type":"ContainerStarted","Data":"7db2d3caa176903b479496fb7ecf0f94675c509d550b201f2f4f46fed9bcf0fc"} Oct 29 23:06:07.715162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:07.715101 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" podStartSLOduration=2.714952243 podStartE2EDuration="2.714952243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:07.711895398 +0000 UTC m=+1364.827815004" watchObservedRunningTime="2026-10-29 23:06:07.714952243 +0000 UTC m=+1364.830871850" Oct 29 23:06:08.252953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:08.252914 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:08.259709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:08.259676 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:08.661762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:08.661671 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:08.662606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:08.662563 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:06:24.213628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:24.213555 5736 scope.go:122] "RemoveContainer" containerID="03efba9318eac56d593c06d808f7ce258c03f9dd545a43fed0c1b326f04d052c" Oct 29 23:06:24.223901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:24.223877 5736 scope.go:122] "RemoveContainer" containerID="5cdf7883b19798b822dffdff191da89f5dac473ef5a665d6d5c0194be7e6da3e" Oct 29 23:06:24.233874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:24.233842 5736 scope.go:122] "RemoveContainer" containerID="9589d963e9a02de71446c51359a9a0d881673c496e0f4b0c9abd3a9bff9d98f7" Oct 29 23:06:26.825471 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:06:26.825436 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:06:27.661836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.661802 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2"] Oct 29 23:06:27.667394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.667373 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.691070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.687505 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2"] Oct 29 23:06:27.698646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.698618 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:06:27.699954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.699935 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:27.745531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.745487 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k49s\" (UniqueName: \"kubernetes.io/projected/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-kube-api-access-6k49s\") pod \"maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.745805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.745682 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.746080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.746052 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.847881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.847847 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k49s\" (UniqueName: \"kubernetes.io/projected/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-kube-api-access-6k49s\") pod \"maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.848359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.847980 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.848359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.848128 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.848633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.848604 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.850559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.850527 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.871203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.871172 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k49s\" (UniqueName: \"kubernetes.io/projected/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-kube-api-access-6k49s\") pod \"maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:27.981545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:27.981522 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:28.288760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:28.288730 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2"] Oct 29 23:06:28.291000 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:06:28.290968 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1abcd2_0577_4ce0_ad2f_9b9f2300562c.slice/crio-aef86be14ea12ca8ec93dd00ccc8c17fe3df25c0efc4c83459524488f99abd44 WatchSource:0}: Error finding container aef86be14ea12ca8ec93dd00ccc8c17fe3df25c0efc4c83459524488f99abd44: Status 404 returned error can't find the container with id aef86be14ea12ca8ec93dd00ccc8c17fe3df25c0efc4c83459524488f99abd44 Oct 29 23:06:28.932194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:28.932106 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" event={"ID":"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c","Type":"ContainerStarted","Data":"0ba6821d67b53effd418464de15c56a760b6de6ea35e2e47aa3a4bd48b8d7797"} Oct 29 23:06:28.932194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:28.932161 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" event={"ID":"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c","Type":"ContainerStarted","Data":"aef86be14ea12ca8ec93dd00ccc8c17fe3df25c0efc4c83459524488f99abd44"} Oct 29 23:06:28.974819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:28.974763 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" podStartSLOduration=1.974748347 podStartE2EDuration="1.974748347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:28.97232023 +0000 UTC m=+1386.088239836" watchObservedRunningTime="2026-10-29 23:06:28.974748347 +0000 UTC m=+1386.090668020" Oct 29 23:06:29.943561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:29.943525 5736 generic.go:356] "Generic (PLEG): container finished" podID="1b1abcd2-0577-4ce0-ad2f-9b9f2300562c" containerID="0ba6821d67b53effd418464de15c56a760b6de6ea35e2e47aa3a4bd48b8d7797" exitCode=0 Oct 29 23:06:29.944316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:29.943633 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" event={"ID":"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c","Type":"ContainerDied","Data":"0ba6821d67b53effd418464de15c56a760b6de6ea35e2e47aa3a4bd48b8d7797"} Oct 29 23:06:31.119139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.119106 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:31.200220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.200178 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-xks-ca\") pod \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " Oct 29 23:06:31.203172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.203089 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-xks-ca" pod "1b1abcd2-0577-4ce0-ad2f-9b9f2300562c" (UID: "1b1abcd2-0577-4ce0-ad2f-9b9f2300562c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:31.235239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.235208 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz"] Oct 29 23:06:31.244017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.243989 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1b1abcd2-0577-4ce0-ad2f-9b9f2300562c" containerName="revoke-keys" Oct 29 23:06:31.244017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.244017 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1abcd2-0577-4ce0-ad2f-9b9f2300562c" containerName="revoke-keys" Oct 29 23:06:31.244372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.244355 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1abcd2-0577-4ce0-ad2f-9b9f2300562c" containerName="revoke-keys" Oct 29 23:06:31.248727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.248706 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.264416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.264388 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz"] Oct 29 23:06:31.302463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.302430 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-kube-api-access-6k49s\" (UniqueName: \"kubernetes.io/projected/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-kube-api-access-6k49s\") pod \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " Oct 29 23:06:31.302650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.302520 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-openshift-ca\") pod \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\" (UID: \"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c\") " Oct 29 23:06:31.304501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.303469 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.304501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.303706 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-openshift-ca" pod "1b1abcd2-0577-4ce0-ad2f-9b9f2300562c" (UID: "1b1abcd2-0577-4ce0-ad2f-9b9f2300562c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:31.309260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.309233 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-kube-api-access-6k49s" pod "1b1abcd2-0577-4ce0-ad2f-9b9f2300562c" (UID: "1b1abcd2-0577-4ce0-ad2f-9b9f2300562c"). InnerVolumeSpecName "kube-api-access-6k49s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:31.404830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.404802 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.404968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.404946 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdr4\" (UniqueName: \"kubernetes.io/projected/c8d81d4d-4b4d-41db-8344-12c9b665a46f-kube-api-access-smdr4\") pod \"maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.405048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.405035 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.405241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.405222 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k49s\" (UniqueName: \"kubernetes.io/projected/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-kube-api-access-6k49s\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.405280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.405244 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.506163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.506133 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.506314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.506231 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdr4\" (UniqueName: \"kubernetes.io/projected/c8d81d4d-4b4d-41db-8344-12c9b665a46f-kube-api-access-smdr4\") pod \"maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.506314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.506289 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.506874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.506849 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.508777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.508753 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.529541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.529519 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdr4\" (UniqueName: \"kubernetes.io/projected/c8d81d4d-4b4d-41db-8344-12c9b665a46f-kube-api-access-smdr4\") pod \"maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.569292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.569271 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:31.885174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.885141 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz"] Oct 29 23:06:31.888229 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:06:31.888197 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d81d4d_4b4d_41db_8344_12c9b665a46f.slice/crio-c4a1ddb2d3973e596e8e62fc0ab53ca6280ff5f030f4c6d80c70e4172c25aac7 WatchSource:0}: Error finding container c4a1ddb2d3973e596e8e62fc0ab53ca6280ff5f030f4c6d80c70e4172c25aac7: Status 404 returned error can't find the container with id c4a1ddb2d3973e596e8e62fc0ab53ca6280ff5f030f4c6d80c70e4172c25aac7 Oct 29 23:06:31.962030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.961996 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" event={"ID":"c8d81d4d-4b4d-41db-8344-12c9b665a46f","Type":"ContainerStarted","Data":"c4a1ddb2d3973e596e8e62fc0ab53ca6280ff5f030f4c6d80c70e4172c25aac7"} Oct 29 23:06:31.963689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.963670 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" Oct 29 23:06:31.963805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.963667 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2" event={"ID":"1b1abcd2-0577-4ce0-ad2f-9b9f2300562c","Type":"ContainerDied","Data":"aef86be14ea12ca8ec93dd00ccc8c17fe3df25c0efc4c83459524488f99abd44"} Oct 29 23:06:31.963805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:31.963801 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aef86be14ea12ca8ec93dd00ccc8c17fe3df25c0efc4c83459524488f99abd44" Oct 29 23:06:32.972722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:32.972687 5736 generic.go:356] "Generic (PLEG): container finished" podID="c8d81d4d-4b4d-41db-8344-12c9b665a46f" containerID="5ef9aa92a978bb7bd704fb53534d686978b02f8fa2186a6c7fa9161a99078730" exitCode=0 Oct 29 23:06:32.973121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:32.972771 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" event={"ID":"c8d81d4d-4b4d-41db-8344-12c9b665a46f","Type":"ContainerDied","Data":"5ef9aa92a978bb7bd704fb53534d686978b02f8fa2186a6c7fa9161a99078730"} Oct 29 23:06:34.278918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.278890 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:34.292863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.292827 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d81d4d-4b4d-41db-8344-12c9b665a46f-kube-api-access-smdr4\" (UniqueName: \"kubernetes.io/projected/c8d81d4d-4b4d-41db-8344-12c9b665a46f-kube-api-access-smdr4\") pod \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " Oct 29 23:06:34.293044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.293025 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-openshift-ca\") pod \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " Oct 29 23:06:34.293260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.293236 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-xks-ca\") pod \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\" (UID: \"c8d81d4d-4b4d-41db-8344-12c9b665a46f\") " Oct 29 23:06:34.293433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.293410 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-openshift-ca" pod "c8d81d4d-4b4d-41db-8344-12c9b665a46f" (UID: "c8d81d4d-4b4d-41db-8344-12c9b665a46f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:34.293963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.293942 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.295376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.295348 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-xks-ca" pod "c8d81d4d-4b4d-41db-8344-12c9b665a46f" (UID: "c8d81d4d-4b4d-41db-8344-12c9b665a46f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:34.295458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.295412 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d81d4d-4b4d-41db-8344-12c9b665a46f-kube-api-access-smdr4" pod "c8d81d4d-4b4d-41db-8344-12c9b665a46f" (UID: "c8d81d4d-4b4d-41db-8344-12c9b665a46f"). InnerVolumeSpecName "kube-api-access-smdr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:34.395210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.395176 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d81d4d-4b4d-41db-8344-12c9b665a46f-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:34.395210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:34.395205 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smdr4\" (UniqueName: \"kubernetes.io/projected/c8d81d4d-4b4d-41db-8344-12c9b665a46f-kube-api-access-smdr4\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:35.058131 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:35.058097 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" event={"ID":"c8d81d4d-4b4d-41db-8344-12c9b665a46f","Type":"ContainerDied","Data":"c4a1ddb2d3973e596e8e62fc0ab53ca6280ff5f030f4c6d80c70e4172c25aac7"} Oct 29 23:06:35.058332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:35.058148 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a1ddb2d3973e596e8e62fc0ab53ca6280ff5f030f4c6d80c70e4172c25aac7" Oct 29 23:06:35.058332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:35.058146 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz" Oct 29 23:06:48.852323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:48.852287 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph"] Oct 29 23:06:48.853309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:48.853287 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c8d81d4d-4b4d-41db-8344-12c9b665a46f" containerName="revoke-keys" Oct 29 23:06:48.853370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:48.853311 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d81d4d-4b4d-41db-8344-12c9b665a46f" containerName="revoke-keys" Oct 29 23:06:48.853517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:48.853503 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d81d4d-4b4d-41db-8344-12c9b665a46f" containerName="revoke-keys" Oct 29 23:06:48.857123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:48.857104 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:06:48.877519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:48.877492 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph"] Oct 29 23:06:48.885619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:48.885563 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cd000a\"" Oct 29 23:06:48.931303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:48.931267 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnhw9\" (UniqueName: \"kubernetes.io/projected/1946032b-9b2d-4b08-abaf-620d8da43823-kube-api-access-xnhw9\") pod \"payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph\" (UID: \"1946032b-9b2d-4b08-abaf-620d8da43823\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:06:48.931459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:48.931401 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1946032b-9b2d-4b08-abaf-620d8da43823-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph\" (UID: \"1946032b-9b2d-4b08-abaf-620d8da43823\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:06:49.045631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:49.032876 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnhw9\" (UniqueName: \"kubernetes.io/projected/1946032b-9b2d-4b08-abaf-620d8da43823-kube-api-access-xnhw9\") pod \"payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph\" (UID: \"1946032b-9b2d-4b08-abaf-620d8da43823\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:06:49.045631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:49.032980 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1946032b-9b2d-4b08-abaf-620d8da43823-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph\" (UID: \"1946032b-9b2d-4b08-abaf-620d8da43823\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:06:49.045631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:49.034026 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1946032b-9b2d-4b08-abaf-620d8da43823-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph\" (UID: \"1946032b-9b2d-4b08-abaf-620d8da43823\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:06:49.234607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:49.232333 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnhw9\" (UniqueName: \"kubernetes.io/projected/1946032b-9b2d-4b08-abaf-620d8da43823-kube-api-access-xnhw9\") pod \"payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph\" (UID: \"1946032b-9b2d-4b08-abaf-620d8da43823\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:06:49.471298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:49.471264 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:06:49.779553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:49.779526 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph"] Oct 29 23:06:49.781798 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:06:49.781768 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1946032b_9b2d_4b08_abaf_620d8da43823.slice/crio-585daadfba1257e51c7065e3764a84ba9c7f8941e6d5dc74e28b80d12de40d8d WatchSource:0}: Error finding container 585daadfba1257e51c7065e3764a84ba9c7f8941e6d5dc74e28b80d12de40d8d: Status 404 returned error can't find the container with id 585daadfba1257e51c7065e3764a84ba9c7f8941e6d5dc74e28b80d12de40d8d Oct 29 23:06:50.289821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:50.289787 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" event={"ID":"1946032b-9b2d-4b08-abaf-620d8da43823","Type":"ContainerStarted","Data":"199a8c0e5163ee851121a4b134b66f6ddfc8d2c65724c8d819cbec9ab73aa8e6"} Oct 29 23:06:50.290196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:50.289837 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" event={"ID":"1946032b-9b2d-4b08-abaf-620d8da43823","Type":"ContainerStarted","Data":"585daadfba1257e51c7065e3764a84ba9c7f8941e6d5dc74e28b80d12de40d8d"} Oct 29 23:06:50.290652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:50.290630 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:06:50.336077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:50.336020 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" podStartSLOduration=2.336005499 podStartE2EDuration="2.336005499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:50.330663009 +0000 UTC m=+1407.446582614" watchObservedRunningTime="2026-10-29 23:06:50.336005499 +0000 UTC m=+1407.451925105" Oct 29 23:06:53.572575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:53.572533 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd"] Oct 29 23:06:53.573052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:53.572805 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" containerID="cri-o://d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4" gracePeriod=30 Oct 29 23:06:56.601761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:56.601715 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh"] Oct 29 23:06:56.602426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:56.601998 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" containerID="cri-o://9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7" gracePeriod=30 Oct 29 23:06:56.828005 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:06:56.827966 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:06:58.864940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.864913 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 23:06:58.952673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.952633 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-kube-api-access-gn46k\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-kube-api-access-gn46k\") pod \"786b0ea4-d2c8-458e-a329-c65354543981\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " Oct 29 23:06:58.952918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.952773 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-data\") pod \"786b0ea4-d2c8-458e-a329-c65354543981\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " Oct 29 23:06:58.952996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.952926 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786b0ea4-d2c8-458e-a329-c65354543981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786b0ea4-d2c8-458e-a329-c65354543981-istiod-ca-cert\") pod \"786b0ea4-d2c8-458e-a329-c65354543981\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " Oct 29 23:06:58.953052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.952996 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-envoy\") pod \"786b0ea4-d2c8-458e-a329-c65354543981\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " Oct 29 23:06:58.953052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.953029 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-istio-token\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-istio-token\") pod \"786b0ea4-d2c8-458e-a329-c65354543981\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " Oct 29 23:06:58.953168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.953058 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-credential-socket\") pod \"786b0ea4-d2c8-458e-a329-c65354543981\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " Oct 29 23:06:58.953168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.953104 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/786b0ea4-d2c8-458e-a329-c65354543981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786b0ea4-d2c8-458e-a329-c65354543981-istio-podinfo\") pod \"786b0ea4-d2c8-458e-a329-c65354543981\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " Oct 29 23:06:58.953168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.953163 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-socket\") pod \"786b0ea4-d2c8-458e-a329-c65354543981\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " Oct 29 23:06:58.953365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.953196 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-certs\") pod \"786b0ea4-d2c8-458e-a329-c65354543981\" (UID: \"786b0ea4-d2c8-458e-a329-c65354543981\") " Oct 29 23:06:58.954221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.954191 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-certs" pod "786b0ea4-d2c8-458e-a329-c65354543981" (UID: "786b0ea4-d2c8-458e-a329-c65354543981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:58.955758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.955731 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-credential-socket" pod "786b0ea4-d2c8-458e-a329-c65354543981" (UID: "786b0ea4-d2c8-458e-a329-c65354543981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:58.956249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.956029 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-envoy" pod "786b0ea4-d2c8-458e-a329-c65354543981" (UID: "786b0ea4-d2c8-458e-a329-c65354543981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:58.956249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.956160 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786b0ea4-d2c8-458e-a329-c65354543981-istiod-ca-cert" pod "786b0ea4-d2c8-458e-a329-c65354543981" (UID: "786b0ea4-d2c8-458e-a329-c65354543981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:58.956539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.956498 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-socket" pod "786b0ea4-d2c8-458e-a329-c65354543981" (UID: "786b0ea4-d2c8-458e-a329-c65354543981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:58.956787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.956762 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-istio-token" pod "786b0ea4-d2c8-458e-a329-c65354543981" (UID: "786b0ea4-d2c8-458e-a329-c65354543981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:58.956876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.956835 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-data" pod "786b0ea4-d2c8-458e-a329-c65354543981" (UID: "786b0ea4-d2c8-458e-a329-c65354543981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:58.960059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.960038 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-kube-api-access-gn46k" pod "786b0ea4-d2c8-458e-a329-c65354543981" (UID: "786b0ea4-d2c8-458e-a329-c65354543981"). InnerVolumeSpecName "kube-api-access-gn46k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:58.961213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:58.961195 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/786b0ea4-d2c8-458e-a329-c65354543981-istio-podinfo" pod "786b0ea4-d2c8-458e-a329-c65354543981" (UID: "786b0ea4-d2c8-458e-a329-c65354543981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:59.056024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.055989 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786b0ea4-d2c8-458e-a329-c65354543981-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:59.056024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.056021 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:59.056024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.056031 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:59.056290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.056040 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:59.056290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.056049 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786b0ea4-d2c8-458e-a329-c65354543981-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:59.056290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.056057 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:59.056290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.056065 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:59.056290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.056074 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn46k\" (UniqueName: \"kubernetes.io/projected/786b0ea4-d2c8-458e-a329-c65354543981-kube-api-access-gn46k\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:59.056290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.056082 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786b0ea4-d2c8-458e-a329-c65354543981-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:06:59.369639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.369561 5736 generic.go:356] "Generic (PLEG): container finished" podID="786b0ea4-d2c8-458e-a329-c65354543981" containerID="d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4" exitCode=0 Oct 29 23:06:59.369936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.369650 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" event={"ID":"786b0ea4-d2c8-458e-a329-c65354543981","Type":"ContainerDied","Data":"d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4"} Oct 29 23:06:59.370032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.369973 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" event={"ID":"786b0ea4-d2c8-458e-a329-c65354543981","Type":"ContainerDied","Data":"af95af4295f3528802396bf2da3c49f9543065f012eea4c5b85ba4bada0afdd9"} Oct 29 23:06:59.370032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.369999 5736 scope.go:122] "RemoveContainer" containerID="d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4" Oct 29 23:06:59.370148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.369667 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd" Oct 29 23:06:59.386910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.386887 5736 scope.go:122] "RemoveContainer" containerID="d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4" Oct 29 23:06:59.387235 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:06:59.387200 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4\": container with ID starting with d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4 not found: ID does not exist" containerID="d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4" Oct 29 23:06:59.387315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.387246 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4"} err="failed to get container status \"d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4\": rpc error: code = NotFound desc = could not find container \"d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4\": container with ID starting with d19686314b83b430709a695ecbbef7e1ba83137fa3be355ae74fcb49461936c4 not found: ID does not exist" Oct 29 23:06:59.436937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.436879 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd"] Oct 29 23:06:59.448262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:06:59.448236 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-b86hd"] Oct 29 23:07:01.005635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.005562 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786b0ea4-d2c8-458e-a329-c65354543981" path="/var/lib/kubelet/pods/786b0ea4-d2c8-458e-a329-c65354543981/volumes" Oct 29 23:07:01.363190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.363106 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2"] Oct 29 23:07:01.373116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.373088 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-44a4e4-f47j2"] Oct 29 23:07:01.901148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.901112 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 23:07:01.940617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.940549 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-credential-socket\") pod \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " Oct 29 23:07:01.940793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.940696 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-certs\") pod \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " Oct 29 23:07:01.940793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.940738 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-envoy\") pod \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " Oct 29 23:07:01.940908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.940794 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istiod-ca-cert\") pod \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " Oct 29 23:07:01.940908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.940804 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-credential-socket" pod "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" (UID: "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.941010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.940928 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-data\") pod \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " Oct 29 23:07:01.941010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.940964 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-socket\") pod \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " Oct 29 23:07:01.941010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.940972 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-certs" pod "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" (UID: "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.941177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941026 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-token\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-token\") pod \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " Oct 29 23:07:01.941177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941093 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-podinfo\") pod \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " Oct 29 23:07:01.941177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941147 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-kube-api-access-vs9gc\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-kube-api-access-vs9gc\") pod \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\" (UID: \"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3\") " Oct 29 23:07:01.941319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941190 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-data" pod "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" (UID: "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.941319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941216 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istiod-ca-cert" pod "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" (UID: "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:01.941319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941225 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-socket" pod "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" (UID: "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.941822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941797 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.941966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941825 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.941966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941840 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.941966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941853 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.941966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.941865 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.944157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.944135 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-envoy" pod "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" (UID: "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.944157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.944150 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-podinfo" pod "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" (UID: "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:01.944308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.944184 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-kube-api-access-vs9gc" pod "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" (UID: "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3"). InnerVolumeSpecName "kube-api-access-vs9gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:01.944308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:01.944210 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-token" pod "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" (UID: "f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:02.043555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.043518 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.043555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.043546 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.043555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.043555 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs9gc\" (UniqueName: \"kubernetes.io/projected/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-kube-api-access-vs9gc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.043555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.043566 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.308850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.308818 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:07:02.490332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.490297 5736 generic.go:356] "Generic (PLEG): container finished" podID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerID="9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7" exitCode=0 Oct 29 23:07:02.490510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.490373 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" Oct 29 23:07:02.490510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.490382 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" event={"ID":"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3","Type":"ContainerDied","Data":"9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7"} Oct 29 23:07:02.490510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.490440 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh" event={"ID":"f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3","Type":"ContainerDied","Data":"cba8a90b8be81709a0eec256abbdc5b9c279ccaa1248f0e834874a0a63a08ab2"} Oct 29 23:07:02.490510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.490473 5736 scope.go:122] "RemoveContainer" containerID="9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7" Oct 29 23:07:02.503925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.503907 5736 scope.go:122] "RemoveContainer" containerID="9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7" Oct 29 23:07:02.504206 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:07:02.504186 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7\": container with ID starting with 9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7 not found: ID does not exist" containerID="9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7" Oct 29 23:07:02.504278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.504213 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7"} err="failed to get container status \"9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7\": rpc error: code = NotFound desc = could not find container \"9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7\": container with ID starting with 9646181d15e1a7fa8ab091a47b31c20969d6577dc2249dd7f087014a6948ebe7 not found: ID does not exist" Oct 29 23:07:02.544597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.544542 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh"] Oct 29 23:07:02.556750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:02.556723 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-b76rh"] Oct 29 23:07:03.005512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:03.005476 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1abcd2-0577-4ce0-ad2f-9b9f2300562c" path="/var/lib/kubelet/pods/1b1abcd2-0577-4ce0-ad2f-9b9f2300562c/volumes" Oct 29 23:07:03.006121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:03.006095 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" path="/var/lib/kubelet/pods/f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3/volumes" Oct 29 23:07:09.076633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.074245 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2"] Oct 29 23:07:09.076633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.074303 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv"] Oct 29 23:07:09.076633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.074480 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" containerID="cri-o://ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145" gracePeriod=30 Oct 29 23:07:09.076633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.074824 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" containerID="cri-o://a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03" gracePeriod=30 Oct 29 23:07:09.623153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.623106 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd"] Oct 29 23:07:09.623786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.623769 5736 cpu_manager.go:414] "containerMap: removing container" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" Oct 29 23:07:09.623845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.623786 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" Oct 29 23:07:09.623845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.623805 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" Oct 29 23:07:09.623845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.623810 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" Oct 29 23:07:09.623947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.623906 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="786b0ea4-d2c8-458e-a329-c65354543981" containerName="istio-proxy" Oct 29 23:07:09.623947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.623916 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58f3e0e-f2c6-4206-b93a-25b3d7e0ffb3" containerName="istio-proxy" Oct 29 23:07:09.629011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.628991 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.661606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.660939 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:07:09.661606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.661121 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:07:09.661989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.661963 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd"] Oct 29 23:07:09.763664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.763625 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.763827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.763666 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwtwm\" (UniqueName: \"kubernetes.io/projected/87b1632c-79ce-43bc-bd58-74f5ad69f21a-kube-api-access-qwtwm\") pod \"maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.763870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.763830 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.869618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.867881 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.869618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.867947 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwtwm\" (UniqueName: \"kubernetes.io/projected/87b1632c-79ce-43bc-bd58-74f5ad69f21a-kube-api-access-qwtwm\") pod \"maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.869618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.868014 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.873031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.872985 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.878938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.878872 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.899188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.899130 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwtwm\" (UniqueName: \"kubernetes.io/projected/87b1632c-79ce-43bc-bd58-74f5ad69f21a-kube-api-access-qwtwm\") pod \"maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:09.976179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:09.976150 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:10.307315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:10.307290 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd"] Oct 29 23:07:10.310200 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:07:10.310168 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b1632c_79ce_43bc_bd58_74f5ad69f21a.slice/crio-81b61a303e5bbffe92d344277b84be85808062840571d50de2e32847deee94f4 WatchSource:0}: Error finding container 81b61a303e5bbffe92d344277b84be85808062840571d50de2e32847deee94f4: Status 404 returned error can't find the container with id 81b61a303e5bbffe92d344277b84be85808062840571d50de2e32847deee94f4 Oct 29 23:07:10.606167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:10.606133 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" event={"ID":"87b1632c-79ce-43bc-bd58-74f5ad69f21a","Type":"ContainerStarted","Data":"81b61a303e5bbffe92d344277b84be85808062840571d50de2e32847deee94f4"} Oct 29 23:07:11.614824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:11.614789 5736 generic.go:356] "Generic (PLEG): container finished" podID="87b1632c-79ce-43bc-bd58-74f5ad69f21a" containerID="4012cf26354b2a99e321a01fff592f1b51f4863a5e30b99249a86bfb8235c725" exitCode=0 Oct 29 23:07:11.615163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:11.614831 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" event={"ID":"87b1632c-79ce-43bc-bd58-74f5ad69f21a","Type":"ContainerDied","Data":"4012cf26354b2a99e321a01fff592f1b51f4863a5e30b99249a86bfb8235c725"} Oct 29 23:07:12.848782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:12.848752 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:12.919259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:12.919182 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-openshift-ca\") pod \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " Oct 29 23:07:12.919437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:12.919415 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-xks-ca\") pod \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " Oct 29 23:07:12.919507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:12.919482 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b1632c-79ce-43bc-bd58-74f5ad69f21a-kube-api-access-qwtwm\" (UniqueName: \"kubernetes.io/projected/87b1632c-79ce-43bc-bd58-74f5ad69f21a-kube-api-access-qwtwm\") pod \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\" (UID: \"87b1632c-79ce-43bc-bd58-74f5ad69f21a\") " Oct 29 23:07:12.920384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:12.920351 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-openshift-ca" pod "87b1632c-79ce-43bc-bd58-74f5ad69f21a" (UID: "87b1632c-79ce-43bc-bd58-74f5ad69f21a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:12.922737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:12.922711 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-xks-ca" pod "87b1632c-79ce-43bc-bd58-74f5ad69f21a" (UID: "87b1632c-79ce-43bc-bd58-74f5ad69f21a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:12.922833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:12.922721 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b1632c-79ce-43bc-bd58-74f5ad69f21a-kube-api-access-qwtwm" pod "87b1632c-79ce-43bc-bd58-74f5ad69f21a" (UID: "87b1632c-79ce-43bc-bd58-74f5ad69f21a"). InnerVolumeSpecName "kube-api-access-qwtwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:13.021401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:13.021366 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:13.021401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:13.021392 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwtwm\" (UniqueName: \"kubernetes.io/projected/87b1632c-79ce-43bc-bd58-74f5ad69f21a-kube-api-access-qwtwm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:13.021562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:13.021405 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87b1632c-79ce-43bc-bd58-74f5ad69f21a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:13.634684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:13.633757 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" event={"ID":"87b1632c-79ce-43bc-bd58-74f5ad69f21a","Type":"ContainerDied","Data":"81b61a303e5bbffe92d344277b84be85808062840571d50de2e32847deee94f4"} Oct 29 23:07:13.634684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:13.633823 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81b61a303e5bbffe92d344277b84be85808062840571d50de2e32847deee94f4" Oct 29 23:07:13.634684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:13.633896 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd" Oct 29 23:07:14.398718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.398684 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:07:14.440895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.440864 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.451429 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-kube-api-access-dwxzb\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-kube-api-access-dwxzb\") pod \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.451489 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-token\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-token\") pod \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.451522 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-certs\") pod \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.451568 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-credential-socket\") pod \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.451656 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-data\") pod \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.451786 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-podinfo\") pod \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.451836 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istiod-ca-cert\") pod \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.451975 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-envoy\") pod \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.452186 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-socket\") pod \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\" (UID: \"fd9be3fc-62e6-4663-abff-5e89dcbfcf46\") " Oct 29 23:07:14.453858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.453525 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-credential-socket" pod "fd9be3fc-62e6-4663-abff-5e89dcbfcf46" (UID: "fd9be3fc-62e6-4663-abff-5e89dcbfcf46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.454601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.454495 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-socket" pod "fd9be3fc-62e6-4663-abff-5e89dcbfcf46" (UID: "fd9be3fc-62e6-4663-abff-5e89dcbfcf46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.455119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.454846 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-data" pod "fd9be3fc-62e6-4663-abff-5e89dcbfcf46" (UID: "fd9be3fc-62e6-4663-abff-5e89dcbfcf46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.455707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.455680 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istiod-ca-cert" pod "fd9be3fc-62e6-4663-abff-5e89dcbfcf46" (UID: "fd9be3fc-62e6-4663-abff-5e89dcbfcf46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:14.456432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.456403 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-certs" pod "fd9be3fc-62e6-4663-abff-5e89dcbfcf46" (UID: "fd9be3fc-62e6-4663-abff-5e89dcbfcf46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.458947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.458917 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-token" pod "fd9be3fc-62e6-4663-abff-5e89dcbfcf46" (UID: "fd9be3fc-62e6-4663-abff-5e89dcbfcf46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:14.460511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.460471 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-podinfo" pod "fd9be3fc-62e6-4663-abff-5e89dcbfcf46" (UID: "fd9be3fc-62e6-4663-abff-5e89dcbfcf46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:14.461010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.460972 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-kube-api-access-dwxzb" pod "fd9be3fc-62e6-4663-abff-5e89dcbfcf46" (UID: "fd9be3fc-62e6-4663-abff-5e89dcbfcf46"). InnerVolumeSpecName "kube-api-access-dwxzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:14.472597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.469206 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-envoy" pod "fd9be3fc-62e6-4663-abff-5e89dcbfcf46" (UID: "fd9be3fc-62e6-4663-abff-5e89dcbfcf46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.554209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.554172 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-kube-api-access-8q72b\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-kube-api-access-8q72b\") pod \"83afa120-ca9b-4466-9323-1668b74fbd48\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " Oct 29 23:07:14.554384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.554224 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-socket\") pod \"83afa120-ca9b-4466-9323-1668b74fbd48\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " Oct 29 23:07:14.554384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.554265 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-certs\") pod \"83afa120-ca9b-4466-9323-1668b74fbd48\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " Oct 29 23:07:14.554519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.554442 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-envoy\") pod \"83afa120-ca9b-4466-9323-1668b74fbd48\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " Oct 29 23:07:14.554519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.554480 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-istio-token\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-istio-token\") pod \"83afa120-ca9b-4466-9323-1668b74fbd48\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " Oct 29 23:07:14.554750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.554571 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-data\") pod \"83afa120-ca9b-4466-9323-1668b74fbd48\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " Oct 29 23:07:14.554750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.554712 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83afa120-ca9b-4466-9323-1668b74fbd48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83afa120-ca9b-4466-9323-1668b74fbd48-istiod-ca-cert\") pod \"83afa120-ca9b-4466-9323-1668b74fbd48\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " Oct 29 23:07:14.554949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.554778 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-credential-socket\") pod \"83afa120-ca9b-4466-9323-1668b74fbd48\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " Oct 29 23:07:14.554949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.554836 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83afa120-ca9b-4466-9323-1668b74fbd48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83afa120-ca9b-4466-9323-1668b74fbd48-istio-podinfo\") pod \"83afa120-ca9b-4466-9323-1668b74fbd48\" (UID: \"83afa120-ca9b-4466-9323-1668b74fbd48\") " Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555386 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-data" pod "83afa120-ca9b-4466-9323-1668b74fbd48" (UID: "83afa120-ca9b-4466-9323-1668b74fbd48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555717 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555743 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwxzb\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-kube-api-access-dwxzb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555760 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555774 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555788 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555803 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555817 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555829 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555842 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.555855 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9be3fc-62e6-4663-abff-5e89dcbfcf46-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.556399 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-credential-socket" pod "83afa120-ca9b-4466-9323-1668b74fbd48" (UID: "83afa120-ca9b-4466-9323-1668b74fbd48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.557026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.556802 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-certs" pod "83afa120-ca9b-4466-9323-1668b74fbd48" (UID: "83afa120-ca9b-4466-9323-1668b74fbd48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.558691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.557003 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-socket" pod "83afa120-ca9b-4466-9323-1668b74fbd48" (UID: "83afa120-ca9b-4466-9323-1668b74fbd48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.558927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.558904 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83afa120-ca9b-4466-9323-1668b74fbd48-istiod-ca-cert" pod "83afa120-ca9b-4466-9323-1668b74fbd48" (UID: "83afa120-ca9b-4466-9323-1668b74fbd48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:14.560719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.560693 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83afa120-ca9b-4466-9323-1668b74fbd48-istio-podinfo" pod "83afa120-ca9b-4466-9323-1668b74fbd48" (UID: "83afa120-ca9b-4466-9323-1668b74fbd48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:14.564203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.564176 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-kube-api-access-8q72b" pod "83afa120-ca9b-4466-9323-1668b74fbd48" (UID: "83afa120-ca9b-4466-9323-1668b74fbd48"). InnerVolumeSpecName "kube-api-access-8q72b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:14.564630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.564573 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-envoy" pod "83afa120-ca9b-4466-9323-1668b74fbd48" (UID: "83afa120-ca9b-4466-9323-1668b74fbd48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:14.566943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.566923 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-istio-token" pod "83afa120-ca9b-4466-9323-1668b74fbd48" (UID: "83afa120-ca9b-4466-9323-1668b74fbd48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:14.644263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.644190 5736 generic.go:356] "Generic (PLEG): container finished" podID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerID="ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145" exitCode=0 Oct 29 23:07:14.644413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.644260 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" Oct 29 23:07:14.644413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.644283 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" event={"ID":"fd9be3fc-62e6-4663-abff-5e89dcbfcf46","Type":"ContainerDied","Data":"ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145"} Oct 29 23:07:14.644413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.644374 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv" event={"ID":"fd9be3fc-62e6-4663-abff-5e89dcbfcf46","Type":"ContainerDied","Data":"a4b70ffc88098c3247602053e3282c22a0186b23ba7e5b2c77c3734a8d68735f"} Oct 29 23:07:14.644413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.644405 5736 scope.go:122] "RemoveContainer" containerID="ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145" Oct 29 23:07:14.645929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.645903 5736 generic.go:356] "Generic (PLEG): container finished" podID="83afa120-ca9b-4466-9323-1668b74fbd48" containerID="a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03" exitCode=0 Oct 29 23:07:14.646037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.645971 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" Oct 29 23:07:14.646037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.645970 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" event={"ID":"83afa120-ca9b-4466-9323-1668b74fbd48","Type":"ContainerDied","Data":"a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03"} Oct 29 23:07:14.646148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.646080 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2" event={"ID":"83afa120-ca9b-4466-9323-1668b74fbd48","Type":"ContainerDied","Data":"72059ed3e1e0366d274c3d164a63eee97642d6b0fe6d127eaa934b6e285afc78"} Oct 29 23:07:14.658831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.658669 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q72b\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-kube-api-access-8q72b\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.658831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.658696 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.658831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.658711 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.658831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.658725 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.658831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.658736 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83afa120-ca9b-4466-9323-1668b74fbd48-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.658831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.658748 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83afa120-ca9b-4466-9323-1668b74fbd48-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.658831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.658759 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83afa120-ca9b-4466-9323-1668b74fbd48-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.658831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.658772 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83afa120-ca9b-4466-9323-1668b74fbd48-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:14.663799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.663767 5736 scope.go:122] "RemoveContainer" containerID="5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266" Oct 29 23:07:14.674653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.674636 5736 scope.go:122] "RemoveContainer" containerID="ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145" Oct 29 23:07:14.674942 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:07:14.674917 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145\": container with ID starting with ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145 not found: ID does not exist" containerID="ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145" Oct 29 23:07:14.674995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.674949 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145"} err="failed to get container status \"ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145\": rpc error: code = NotFound desc = could not find container \"ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145\": container with ID starting with ec4bfc2a6b69d98c945802492b99c1682c4c5dc26cf6a52976d0b7a84a3ad145 not found: ID does not exist" Oct 29 23:07:14.674995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.674966 5736 scope.go:122] "RemoveContainer" containerID="5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266" Oct 29 23:07:14.675231 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:07:14.675216 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266\": container with ID starting with 5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266 not found: ID does not exist" containerID="5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266" Oct 29 23:07:14.675278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.675236 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266"} err="failed to get container status \"5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266\": rpc error: code = NotFound desc = could not find container \"5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266\": container with ID starting with 5f98451b15cbb09ea04b4ead932f1bcd85201f081c20bb3d76d4a6a2403b3266 not found: ID does not exist" Oct 29 23:07:14.675278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.675250 5736 scope.go:122] "RemoveContainer" containerID="a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03" Oct 29 23:07:14.685047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.685023 5736 scope.go:122] "RemoveContainer" containerID="a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03" Oct 29 23:07:14.685355 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:07:14.685331 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03\": container with ID starting with a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03 not found: ID does not exist" containerID="a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03" Oct 29 23:07:14.685422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.685366 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03"} err="failed to get container status \"a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03\": rpc error: code = NotFound desc = could not find container \"a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03\": container with ID starting with a2140ae7b7d6f5daba0a50f114c94f7ca837aa30c65160f9ec9abf4ea5b1bd03 not found: ID does not exist" Oct 29 23:07:14.707973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.707942 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv"] Oct 29 23:07:14.720535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.720486 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-cfdlv"] Oct 29 23:07:14.723552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.723530 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx"] Oct 29 23:07:14.724155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724140 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" Oct 29 23:07:14.724201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724158 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" Oct 29 23:07:14.724201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724193 5736 cpu_manager.go:414] "containerMap: removing container" podUID="87b1632c-79ce-43bc-bd58-74f5ad69f21a" containerName="revoke-keys" Oct 29 23:07:14.724201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724198 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b1632c-79ce-43bc-bd58-74f5ad69f21a" containerName="revoke-keys" Oct 29 23:07:14.724294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724210 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" Oct 29 23:07:14.724294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724215 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" Oct 29 23:07:14.724294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724229 5736 cpu_manager.go:414] "containerMap: removing container" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" Oct 29 23:07:14.724294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724233 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" Oct 29 23:07:14.724413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724348 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b1632c-79ce-43bc-bd58-74f5ad69f21a" containerName="revoke-keys" Oct 29 23:07:14.724413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724357 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" Oct 29 23:07:14.724413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.724365 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" containerName="istio-proxy" Oct 29 23:07:14.729681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.729651 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.747747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.747718 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx"] Oct 29 23:07:14.748015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.747996 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7af1a5\"/\"kube-root-ca.crt\"" Oct 29 23:07:14.748114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.748100 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7af1a5\"/\"default-dockercfg-9qcwb\"" Oct 29 23:07:14.748224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.748209 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7af1a5\"/\"ipp-route-7af1a5-kserve-self-signed-certs\"" Oct 29 23:07:14.748309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.748295 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7af1a5\"/\"openshift-service-ca.crt\"" Oct 29 23:07:14.757693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.757635 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2"] Oct 29 23:07:14.761635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.761567 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-jt5s2"] Oct 29 23:07:14.872673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.861978 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-dshm\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.872673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.862025 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db18ea92-9d40-4840-9c15-353d841dea31-tls-certs\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.872673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.862066 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-model-cache\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.872673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.862098 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-home\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.872673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.862149 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dfr\" (UniqueName: \"kubernetes.io/projected/db18ea92-9d40-4840-9c15-353d841dea31-kube-api-access-b6dfr\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.872673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.862202 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-tmp-dir\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.964407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.963814 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-dshm\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.964407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.963864 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db18ea92-9d40-4840-9c15-353d841dea31-tls-certs\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.964407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.963904 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-model-cache\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.964407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.963935 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-home\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.964407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.963986 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dfr\" (UniqueName: \"kubernetes.io/projected/db18ea92-9d40-4840-9c15-353d841dea31-kube-api-access-b6dfr\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.964407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.964041 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-tmp-dir\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.964903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.964535 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-model-cache\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.964903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.964626 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-tmp-dir\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.964903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.964845 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-home\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.967698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.967007 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-dshm\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.967698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.967481 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db18ea92-9d40-4840-9c15-353d841dea31-tls-certs\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:14.999528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:14.990628 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dfr\" (UniqueName: \"kubernetes.io/projected/db18ea92-9d40-4840-9c15-353d841dea31-kube-api-access-b6dfr\") pod \"ipp-route-7af1a5-kserve-5555f84b85-fcdpx\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:15.058606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:15.057856 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:15.092450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:15.070194 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83afa120-ca9b-4466-9323-1668b74fbd48" path="/var/lib/kubelet/pods/83afa120-ca9b-4466-9323-1668b74fbd48/volumes" Oct 29 23:07:15.099603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:15.094743 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" path="/var/lib/kubelet/pods/fd9be3fc-62e6-4663-abff-5e89dcbfcf46/volumes" Oct 29 23:07:15.323207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:15.323160 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2"] Oct 29 23:07:15.323482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:15.323447 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" containerID="cri-o://15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6" gracePeriod=30 Oct 29 23:07:15.498681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:15.493906 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx"] Oct 29 23:07:15.505331 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:07:15.505074 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb18ea92_9d40_4840_9c15_353d841dea31.slice/crio-a15f4a549ef3582041574a336126f0f9fd98228acdb61e504ea96f07072ff28a WatchSource:0}: Error finding container a15f4a549ef3582041574a336126f0f9fd98228acdb61e504ea96f07072ff28a: Status 404 returned error can't find the container with id a15f4a549ef3582041574a336126f0f9fd98228acdb61e504ea96f07072ff28a Oct 29 23:07:15.659406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:15.659359 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" event={"ID":"db18ea92-9d40-4840-9c15-353d841dea31","Type":"ContainerStarted","Data":"8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace"} Oct 29 23:07:15.659637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:15.659424 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" event={"ID":"db18ea92-9d40-4840-9c15-353d841dea31","Type":"ContainerStarted","Data":"a15f4a549ef3582041574a336126f0f9fd98228acdb61e504ea96f07072ff28a"} Oct 29 23:07:15.703541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:15.703488 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" podStartSLOduration=1.703471714 podStartE2EDuration="1.703471714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:15.700667456 +0000 UTC m=+1432.816587062" watchObservedRunningTime="2026-10-29 23:07:15.703471714 +0000 UTC m=+1432.819391321" Oct 29 23:07:16.058725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:16.058687 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:17.734915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:17.733248 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": read tcp 10.134.0.2:53234->10.134.0.156:15021: read: connection reset by peer" start-of-body= Oct 29 23:07:17.734915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:17.733315 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": read tcp 10.134.0.2:53234->10.134.0.156:15021: read: connection reset by peer" Oct 29 23:07:18.687145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:18.687113 5736 generic.go:356] "Generic (PLEG): container finished" podID="f79fd910-a277-4b22-849e-bead58762990" containerID="cb80dd9eb8bec558e57f778e4f30a14b216e33c397a2f790ed61b84f3e297f24" exitCode=0 Oct 29 23:07:18.687318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:18.687157 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" event={"ID":"f79fd910-a277-4b22-849e-bead58762990","Type":"ContainerDied","Data":"cb80dd9eb8bec558e57f778e4f30a14b216e33c397a2f790ed61b84f3e297f24"} Oct 29 23:07:18.687536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:18.687522 5736 scope.go:122] "RemoveContainer" containerID="cb80dd9eb8bec558e57f778e4f30a14b216e33c397a2f790ed61b84f3e297f24" Oct 29 23:07:18.689225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:18.689195 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:07:18.689301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:18.689268 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:07:18.689339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:18.689300 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:07:19.696758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:19.696720 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" event={"ID":"f79fd910-a277-4b22-849e-bead58762990","Type":"ContainerStarted","Data":"0e84d65f4008c2a67da589e991e2108879ad494c7e48d2d1ff744df76a70d084"} Oct 29 23:07:20.580903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.580819 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:07:20.581138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.581008 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:20.581138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.581057 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 23:07:20.664932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.664905 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 23:07:20.706004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.705968 5736 generic.go:356] "Generic (PLEG): container finished" podID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerID="15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6" exitCode=0 Oct 29 23:07:20.706391 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.706008 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" Oct 29 23:07:20.706391 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.706057 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" event={"ID":"192952f8-2bda-4164-9e1a-5dbbfbc63017","Type":"ContainerDied","Data":"15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6"} Oct 29 23:07:20.706391 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.706102 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2" event={"ID":"192952f8-2bda-4164-9e1a-5dbbfbc63017","Type":"ContainerDied","Data":"c3207d9a49d8a8db536ead552fda494ed6cb2a5304cb4bf8b771441b41348e4c"} Oct 29 23:07:20.706391 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.706127 5736 scope.go:122] "RemoveContainer" containerID="15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6" Oct 29 23:07:20.719674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.719652 5736 scope.go:122] "RemoveContainer" containerID="15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6" Oct 29 23:07:20.723358 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:07:20.723317 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6\": container with ID starting with 15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6 not found: ID does not exist" containerID="15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6" Oct 29 23:07:20.723554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.723364 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6"} err="failed to get container status \"15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6\": rpc error: code = NotFound desc = could not find container \"15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6\": container with ID starting with 15964b98e2db84c449916e936fc09af4da1a54627439b1380f77a4ea23060da6 not found: ID does not exist" Oct 29 23:07:20.801067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801035 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-socket\") pod \"192952f8-2bda-4164-9e1a-5dbbfbc63017\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " Oct 29 23:07:20.801267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801079 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-token\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-token\") pod \"192952f8-2bda-4164-9e1a-5dbbfbc63017\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " Oct 29 23:07:20.801267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801112 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-certs\") pod \"192952f8-2bda-4164-9e1a-5dbbfbc63017\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " Oct 29 23:07:20.801267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801159 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-kube-api-access-sm5b9\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-kube-api-access-sm5b9\") pod \"192952f8-2bda-4164-9e1a-5dbbfbc63017\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " Oct 29 23:07:20.801432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801272 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-socket" pod "192952f8-2bda-4164-9e1a-5dbbfbc63017" (UID: "192952f8-2bda-4164-9e1a-5dbbfbc63017"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:20.801432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801344 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-podinfo\") pod \"192952f8-2bda-4164-9e1a-5dbbfbc63017\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " Oct 29 23:07:20.801432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801378 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-envoy\") pod \"192952f8-2bda-4164-9e1a-5dbbfbc63017\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " Oct 29 23:07:20.801600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801439 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-data\") pod \"192952f8-2bda-4164-9e1a-5dbbfbc63017\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " Oct 29 23:07:20.802141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801685 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-credential-socket\") pod \"192952f8-2bda-4164-9e1a-5dbbfbc63017\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " Oct 29 23:07:20.802141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801728 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192952f8-2bda-4164-9e1a-5dbbfbc63017-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192952f8-2bda-4164-9e1a-5dbbfbc63017-istiod-ca-cert\") pod \"192952f8-2bda-4164-9e1a-5dbbfbc63017\" (UID: \"192952f8-2bda-4164-9e1a-5dbbfbc63017\") " Oct 29 23:07:20.802141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.801915 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-certs" pod "192952f8-2bda-4164-9e1a-5dbbfbc63017" (UID: "192952f8-2bda-4164-9e1a-5dbbfbc63017"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:20.802141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.802078 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-data" pod "192952f8-2bda-4164-9e1a-5dbbfbc63017" (UID: "192952f8-2bda-4164-9e1a-5dbbfbc63017"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:20.802671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.802246 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-credential-socket" pod "192952f8-2bda-4164-9e1a-5dbbfbc63017" (UID: "192952f8-2bda-4164-9e1a-5dbbfbc63017"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:20.802671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.802312 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192952f8-2bda-4164-9e1a-5dbbfbc63017-istiod-ca-cert" pod "192952f8-2bda-4164-9e1a-5dbbfbc63017" (UID: "192952f8-2bda-4164-9e1a-5dbbfbc63017"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:20.803123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.803033 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.803123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.803061 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.803123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.803075 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192952f8-2bda-4164-9e1a-5dbbfbc63017-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.803123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.803088 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.803123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.803102 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.803572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.803553 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-kube-api-access-sm5b9" pod "192952f8-2bda-4164-9e1a-5dbbfbc63017" (UID: "192952f8-2bda-4164-9e1a-5dbbfbc63017"). InnerVolumeSpecName "kube-api-access-sm5b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:20.803852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.803829 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-envoy" pod "192952f8-2bda-4164-9e1a-5dbbfbc63017" (UID: "192952f8-2bda-4164-9e1a-5dbbfbc63017"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:20.804706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.804684 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-podinfo" pod "192952f8-2bda-4164-9e1a-5dbbfbc63017" (UID: "192952f8-2bda-4164-9e1a-5dbbfbc63017"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:20.805035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.805012 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-token" pod "192952f8-2bda-4164-9e1a-5dbbfbc63017" (UID: "192952f8-2bda-4164-9e1a-5dbbfbc63017"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:20.905162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.905074 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.905162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.905108 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm5b9\" (UniqueName: \"kubernetes.io/projected/192952f8-2bda-4164-9e1a-5dbbfbc63017-kube-api-access-sm5b9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.905162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.905122 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.905162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:20.905142 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192952f8-2bda-4164-9e1a-5dbbfbc63017-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:21.077068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:21.077035 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2"] Oct 29 23:07:21.087888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:21.087863 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-vqtn2"] Oct 29 23:07:21.579009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:21.578975 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:21.579177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:21.579040 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 23:07:22.579223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:22.579194 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:22.579621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:22.579253 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 23:07:23.007046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:23.005712 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" path="/var/lib/kubelet/pods/192952f8-2bda-4164-9e1a-5dbbfbc63017/volumes" Oct 29 23:07:23.579569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:23.579536 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:23.580059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:23.579627 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 23:07:24.288940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:24.287087 5736 scope.go:122] "RemoveContainer" containerID="44af1f18ec3b5f85633a8d930908b2fa2042c3743b895f8edc861450f8d82dc2" Oct 29 23:07:24.578705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:24.578625 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 23:07:24.578705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:24.578681 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 23:07:25.070604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:25.062522 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:25.148607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:25.146727 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:25.583935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:25.583906 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:07:25.584200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:25.584179 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:07:25.585075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:25.585055 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:07:25.779441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:25.779412 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:07:26.834221 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:07:26.834172 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:07:30.353124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:30.353064 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd"] Oct 29 23:07:30.375604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:30.374270 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-41b12c-sn2nd"] Oct 29 23:07:31.005974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:31.005938 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b1632c-79ce-43bc-bd58-74f5ad69f21a" path="/var/lib/kubelet/pods/87b1632c-79ce-43bc-bd58-74f5ad69f21a/volumes" Oct 29 23:07:39.226352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:39.226267 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz"] Oct 29 23:07:39.239357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:39.239315 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b0c72c-jq5kz"] Oct 29 23:07:39.991572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:39.991534 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p"] Oct 29 23:07:39.992456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:39.992430 5736 cpu_manager.go:414] "containerMap: removing container" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" Oct 29 23:07:39.992456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:39.992452 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" Oct 29 23:07:39.992738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:39.992676 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="192952f8-2bda-4164-9e1a-5dbbfbc63017" containerName="istio-proxy" Oct 29 23:07:39.992738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:39.992696 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9be3fc-62e6-4663-abff-5e89dcbfcf46" containerName="istio-proxy" Oct 29 23:07:39.998383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:39.998356 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.021213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.021186 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p"] Oct 29 23:07:40.029365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.029340 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:07:40.029544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.029374 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:07:40.051739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.051711 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.051891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.051780 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.051891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.051879 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6vs\" (UniqueName: \"kubernetes.io/projected/2871a25b-e0ad-4b03-ac69-fb11942c43c5-kube-api-access-dv6vs\") pod \"maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.153486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.153407 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.153821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.153505 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.153821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.153536 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6vs\" (UniqueName: \"kubernetes.io/projected/2871a25b-e0ad-4b03-ac69-fb11942c43c5-kube-api-access-dv6vs\") pod \"maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.195815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.195779 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.196227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.196198 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6vs\" (UniqueName: \"kubernetes.io/projected/2871a25b-e0ad-4b03-ac69-fb11942c43c5-kube-api-access-dv6vs\") pod \"maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.226106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.226068 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.313939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.313809 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:40.765866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.763282 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p"] Oct 29 23:07:40.773424 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:07:40.773389 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2871a25b_e0ad_4b03_ac69_fb11942c43c5.slice/crio-fc246c1131450e38f8eb0715b26c413f9d93e5615d1b2200dbcab70c8fbce02a WatchSource:0}: Error finding container fc246c1131450e38f8eb0715b26c413f9d93e5615d1b2200dbcab70c8fbce02a: Status 404 returned error can't find the container with id fc246c1131450e38f8eb0715b26c413f9d93e5615d1b2200dbcab70c8fbce02a Oct 29 23:07:40.977706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:40.977671 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" event={"ID":"2871a25b-e0ad-4b03-ac69-fb11942c43c5","Type":"ContainerStarted","Data":"fc246c1131450e38f8eb0715b26c413f9d93e5615d1b2200dbcab70c8fbce02a"} Oct 29 23:07:41.005155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:41.005111 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d81d4d-4b4d-41db-8344-12c9b665a46f" path="/var/lib/kubelet/pods/c8d81d4d-4b4d-41db-8344-12c9b665a46f/volumes" Oct 29 23:07:41.989492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:41.989459 5736 generic.go:356] "Generic (PLEG): container finished" podID="2871a25b-e0ad-4b03-ac69-fb11942c43c5" containerID="42ffb3cdd15c429513503ba88cccb9a154beaef60c3b6094c0c2593f85d50c1c" exitCode=0 Oct 29 23:07:41.990057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:41.989541 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" event={"ID":"2871a25b-e0ad-4b03-ac69-fb11942c43c5","Type":"ContainerDied","Data":"42ffb3cdd15c429513503ba88cccb9a154beaef60c3b6094c0c2593f85d50c1c"} Oct 29 23:07:43.163126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.163100 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:43.322155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.322077 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-openshift-ca\") pod \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " Oct 29 23:07:43.322155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.322146 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2871a25b-e0ad-4b03-ac69-fb11942c43c5-kube-api-access-dv6vs\" (UniqueName: \"kubernetes.io/projected/2871a25b-e0ad-4b03-ac69-fb11942c43c5-kube-api-access-dv6vs\") pod \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " Oct 29 23:07:43.322394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.322181 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-xks-ca\") pod \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\" (UID: \"2871a25b-e0ad-4b03-ac69-fb11942c43c5\") " Oct 29 23:07:43.322460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.322430 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-openshift-ca" pod "2871a25b-e0ad-4b03-ac69-fb11942c43c5" (UID: "2871a25b-e0ad-4b03-ac69-fb11942c43c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:43.322745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.322723 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:43.324244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.324218 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-xks-ca" pod "2871a25b-e0ad-4b03-ac69-fb11942c43c5" (UID: "2871a25b-e0ad-4b03-ac69-fb11942c43c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:43.324333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.324243 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2871a25b-e0ad-4b03-ac69-fb11942c43c5-kube-api-access-dv6vs" pod "2871a25b-e0ad-4b03-ac69-fb11942c43c5" (UID: "2871a25b-e0ad-4b03-ac69-fb11942c43c5"). InnerVolumeSpecName "kube-api-access-dv6vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:43.425618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.424842 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2871a25b-e0ad-4b03-ac69-fb11942c43c5-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:43.425618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:43.424876 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6vs\" (UniqueName: \"kubernetes.io/projected/2871a25b-e0ad-4b03-ac69-fb11942c43c5-kube-api-access-dv6vs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:44.005997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:44.005970 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" Oct 29 23:07:44.006211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:44.005994 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p" event={"ID":"2871a25b-e0ad-4b03-ac69-fb11942c43c5","Type":"ContainerDied","Data":"fc246c1131450e38f8eb0715b26c413f9d93e5615d1b2200dbcab70c8fbce02a"} Oct 29 23:07:44.006211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:44.006036 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc246c1131450e38f8eb0715b26c413f9d93e5615d1b2200dbcab70c8fbce02a" Oct 29 23:07:47.382647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:47.380898 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk"] Oct 29 23:07:47.382647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:47.381567 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerName="istio-proxy" containerID="cri-o://db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92" gracePeriod=30 Oct 29 23:07:47.390443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:47.390415 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth"] Oct 29 23:07:47.390660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:47.390636 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" containerID="cri-o://608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c" gracePeriod=30 Oct 29 23:07:47.875262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:47.875227 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn"] Oct 29 23:07:47.875444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:47.875422 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" containerID="cri-o://736421b4c3fc58fa47af1147a92388e1fbfc2b2cd19655bf046c1f79d5b4703e" gracePeriod=30 Oct 29 23:07:48.000351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.000313 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9"] Oct 29 23:07:48.001145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.001119 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2871a25b-e0ad-4b03-ac69-fb11942c43c5" containerName="revoke-keys" Oct 29 23:07:48.001200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.001146 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2871a25b-e0ad-4b03-ac69-fb11942c43c5" containerName="revoke-keys" Oct 29 23:07:48.001305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.001296 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2871a25b-e0ad-4b03-ac69-fb11942c43c5" containerName="revoke-keys" Oct 29 23:07:48.005039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.005017 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.020559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.020534 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:07:48.020559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.020541 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:07:48.024529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.024504 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9"] Oct 29 23:07:48.123685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.123652 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhmf8\" (UniqueName: \"kubernetes.io/projected/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-kube-api-access-zhmf8\") pod \"maas-api-revoke-keys-e2e-api-b-57b28e-c67b9\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.123845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.123781 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57b28e-c67b9\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.123975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.123947 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57b28e-c67b9\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.225356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.225329 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57b28e-c67b9\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.225525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.225445 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhmf8\" (UniqueName: \"kubernetes.io/projected/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-kube-api-access-zhmf8\") pod \"maas-api-revoke-keys-e2e-api-b-57b28e-c67b9\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.225525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.225509 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57b28e-c67b9\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.226098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.226071 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57b28e-c67b9\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.227930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.227909 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57b28e-c67b9\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.248277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.248251 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhmf8\" (UniqueName: \"kubernetes.io/projected/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-kube-api-access-zhmf8\") pod \"maas-api-revoke-keys-e2e-api-b-57b28e-c67b9\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.319357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.319335 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:48.683899 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:48.683851 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9"] Oct 29 23:07:48.684845 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:07:48.684818 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b62de84_4b61_44e4_801f_48f3c7b7fbb4.slice/crio-da1799b378eea8bf5d2a54523dd9ff2ebeeb6ade677a30d8600be8569008bbda WatchSource:0}: Error finding container da1799b378eea8bf5d2a54523dd9ff2ebeeb6ade677a30d8600be8569008bbda: Status 404 returned error can't find the container with id da1799b378eea8bf5d2a54523dd9ff2ebeeb6ade677a30d8600be8569008bbda Oct 29 23:07:49.046951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:49.046921 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" event={"ID":"3b62de84-4b61-44e4-801f-48f3c7b7fbb4","Type":"ContainerStarted","Data":"da1799b378eea8bf5d2a54523dd9ff2ebeeb6ade677a30d8600be8569008bbda"} Oct 29 23:07:50.066562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:50.066524 5736 generic.go:356] "Generic (PLEG): container finished" podID="3b62de84-4b61-44e4-801f-48f3c7b7fbb4" containerID="1c8a74b3160ea11a37ee78ffe2bf42a6de704b8c2563ba50f69d26f2bc5ef9fa" exitCode=0 Oct 29 23:07:50.067396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:50.067367 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" event={"ID":"3b62de84-4b61-44e4-801f-48f3c7b7fbb4","Type":"ContainerDied","Data":"1c8a74b3160ea11a37ee78ffe2bf42a6de704b8c2563ba50f69d26f2bc5ef9fa"} Oct 29 23:07:50.987538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:50.987504 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:07:50.987732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:50.987589 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:07:51.427997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.427965 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:51.517218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.517186 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-xks-ca\") pod \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " Oct 29 23:07:51.517384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.517228 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-kube-api-access-zhmf8\" (UniqueName: \"kubernetes.io/projected/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-kube-api-access-zhmf8\") pod \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " Oct 29 23:07:51.517455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.517442 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-openshift-ca\") pod \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\" (UID: \"3b62de84-4b61-44e4-801f-48f3c7b7fbb4\") " Oct 29 23:07:51.520604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.518345 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-openshift-ca" pod "3b62de84-4b61-44e4-801f-48f3c7b7fbb4" (UID: "3b62de84-4b61-44e4-801f-48f3c7b7fbb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:51.527908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.527814 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-xks-ca" pod "3b62de84-4b61-44e4-801f-48f3c7b7fbb4" (UID: "3b62de84-4b61-44e4-801f-48f3c7b7fbb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:51.528356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.528338 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-kube-api-access-zhmf8" pod "3b62de84-4b61-44e4-801f-48f3c7b7fbb4" (UID: "3b62de84-4b61-44e4-801f-48f3c7b7fbb4"). InnerVolumeSpecName "kube-api-access-zhmf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:51.619452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.619416 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:51.619452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.619445 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:51.619452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:51.619456 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhmf8\" (UniqueName: \"kubernetes.io/projected/3b62de84-4b61-44e4-801f-48f3c7b7fbb4-kube-api-access-zhmf8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.085753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.085721 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" event={"ID":"3b62de84-4b61-44e4-801f-48f3c7b7fbb4","Type":"ContainerDied","Data":"da1799b378eea8bf5d2a54523dd9ff2ebeeb6ade677a30d8600be8569008bbda"} Oct 29 23:07:52.085753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.085751 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9" Oct 29 23:07:52.085939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.085769 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da1799b378eea8bf5d2a54523dd9ff2ebeeb6ade677a30d8600be8569008bbda" Oct 29 23:07:52.802508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.802472 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:07:52.865203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.865165 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-socket\") pod \"42ddbbdc-958f-44f6-bf08-aad1877e8875\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " Oct 29 23:07:52.865364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.865336 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-data\") pod \"42ddbbdc-958f-44f6-bf08-aad1877e8875\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " Oct 29 23:07:52.865431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.865404 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ddbbdc-958f-44f6-bf08-aad1877e8875-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddbbdc-958f-44f6-bf08-aad1877e8875-istiod-ca-cert\") pod \"42ddbbdc-958f-44f6-bf08-aad1877e8875\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " Oct 29 23:07:52.865489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.865450 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-token\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-token\") pod \"42ddbbdc-958f-44f6-bf08-aad1877e8875\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " Oct 29 23:07:52.865664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.865642 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-podinfo\") pod \"42ddbbdc-958f-44f6-bf08-aad1877e8875\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " Oct 29 23:07:52.865741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.865691 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-credential-socket\") pod \"42ddbbdc-958f-44f6-bf08-aad1877e8875\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " Oct 29 23:07:52.865813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.865797 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-envoy\") pod \"42ddbbdc-958f-44f6-bf08-aad1877e8875\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " Oct 29 23:07:52.865880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.865863 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-certs\") pod \"42ddbbdc-958f-44f6-bf08-aad1877e8875\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " Oct 29 23:07:52.866042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.865976 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-kube-api-access-9ctqp\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-kube-api-access-9ctqp\") pod \"42ddbbdc-958f-44f6-bf08-aad1877e8875\" (UID: \"42ddbbdc-958f-44f6-bf08-aad1877e8875\") " Oct 29 23:07:52.868774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.868358 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-data" pod "42ddbbdc-958f-44f6-bf08-aad1877e8875" (UID: "42ddbbdc-958f-44f6-bf08-aad1877e8875"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:52.868774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.868527 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-certs" pod "42ddbbdc-958f-44f6-bf08-aad1877e8875" (UID: "42ddbbdc-958f-44f6-bf08-aad1877e8875"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:52.868774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.868570 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ddbbdc-958f-44f6-bf08-aad1877e8875-istiod-ca-cert" pod "42ddbbdc-958f-44f6-bf08-aad1877e8875" (UID: "42ddbbdc-958f-44f6-bf08-aad1877e8875"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:52.869111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.869058 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-credential-socket" pod "42ddbbdc-958f-44f6-bf08-aad1877e8875" (UID: "42ddbbdc-958f-44f6-bf08-aad1877e8875"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:52.869801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.869780 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-socket" pod "42ddbbdc-958f-44f6-bf08-aad1877e8875" (UID: "42ddbbdc-958f-44f6-bf08-aad1877e8875"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:52.872364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.872195 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-podinfo" pod "42ddbbdc-958f-44f6-bf08-aad1877e8875" (UID: "42ddbbdc-958f-44f6-bf08-aad1877e8875"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:52.872364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.872332 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-kube-api-access-9ctqp" pod "42ddbbdc-958f-44f6-bf08-aad1877e8875" (UID: "42ddbbdc-958f-44f6-bf08-aad1877e8875"). InnerVolumeSpecName "kube-api-access-9ctqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:52.872973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.872940 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-envoy" pod "42ddbbdc-958f-44f6-bf08-aad1877e8875" (UID: "42ddbbdc-958f-44f6-bf08-aad1877e8875"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:52.875012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.874981 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-token" pod "42ddbbdc-958f-44f6-bf08-aad1877e8875" (UID: "42ddbbdc-958f-44f6-bf08-aad1877e8875"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:52.912168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.911811 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.976186 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-kube-api-access-jqnqf\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-kube-api-access-jqnqf\") pod \"024e7072-b667-4a09-ab8e-6eadbce1c464\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.976257 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-socket\") pod \"024e7072-b667-4a09-ab8e-6eadbce1c464\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.976305 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-token\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-token\") pod \"024e7072-b667-4a09-ab8e-6eadbce1c464\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.976365 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-data\") pod \"024e7072-b667-4a09-ab8e-6eadbce1c464\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.976462 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-certs\") pod \"024e7072-b667-4a09-ab8e-6eadbce1c464\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.976598 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-envoy\") pod \"024e7072-b667-4a09-ab8e-6eadbce1c464\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.976703 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024e7072-b667-4a09-ab8e-6eadbce1c464-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024e7072-b667-4a09-ab8e-6eadbce1c464-istiod-ca-cert\") pod \"024e7072-b667-4a09-ab8e-6eadbce1c464\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.976731 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-podinfo\") pod \"024e7072-b667-4a09-ab8e-6eadbce1c464\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.976867 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-credential-socket\") pod \"024e7072-b667-4a09-ab8e-6eadbce1c464\" (UID: \"024e7072-b667-4a09-ab8e-6eadbce1c464\") " Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.977678 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.977697 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctqp\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-kube-api-access-9ctqp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.977712 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.977737 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.977751 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ddbbdc-958f-44f6-bf08-aad1877e8875-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.977764 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.977776 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.977788 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.977813 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ddbbdc-958f-44f6-bf08-aad1877e8875-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.978082 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-credential-socket" pod "024e7072-b667-4a09-ab8e-6eadbce1c464" (UID: "024e7072-b667-4a09-ab8e-6eadbce1c464"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.980113 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-socket" pod "024e7072-b667-4a09-ab8e-6eadbce1c464" (UID: "024e7072-b667-4a09-ab8e-6eadbce1c464"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:52.980969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.980297 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-data" pod "024e7072-b667-4a09-ab8e-6eadbce1c464" (UID: "024e7072-b667-4a09-ab8e-6eadbce1c464"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:52.982191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.981056 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024e7072-b667-4a09-ab8e-6eadbce1c464-istiod-ca-cert" pod "024e7072-b667-4a09-ab8e-6eadbce1c464" (UID: "024e7072-b667-4a09-ab8e-6eadbce1c464"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:52.982191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.981317 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-certs" pod "024e7072-b667-4a09-ab8e-6eadbce1c464" (UID: "024e7072-b667-4a09-ab8e-6eadbce1c464"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:52.982191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.981826 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-kube-api-access-jqnqf" pod "024e7072-b667-4a09-ab8e-6eadbce1c464" (UID: "024e7072-b667-4a09-ab8e-6eadbce1c464"). InnerVolumeSpecName "kube-api-access-jqnqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:52.991725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:52.991287 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-token" pod "024e7072-b667-4a09-ab8e-6eadbce1c464" (UID: "024e7072-b667-4a09-ab8e-6eadbce1c464"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:53.020643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.018561 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-envoy" pod "024e7072-b667-4a09-ab8e-6eadbce1c464" (UID: "024e7072-b667-4a09-ab8e-6eadbce1c464"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.022053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.022020 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-podinfo" pod "024e7072-b667-4a09-ab8e-6eadbce1c464" (UID: "024e7072-b667-4a09-ab8e-6eadbce1c464"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:53.079508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.079378 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.079508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.079407 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.079508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.079416 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/024e7072-b667-4a09-ab8e-6eadbce1c464-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.079508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.079423 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.079508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.079435 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.079508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.079448 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqnqf\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-kube-api-access-jqnqf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.079508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.079457 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.079508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.079466 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.079508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.079477 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/024e7072-b667-4a09-ab8e-6eadbce1c464-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.098105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.098062 5736 generic.go:356] "Generic (PLEG): container finished" podID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerID="db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92" exitCode=0 Oct 29 23:07:53.098292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.098210 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" event={"ID":"42ddbbdc-958f-44f6-bf08-aad1877e8875","Type":"ContainerDied","Data":"db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92"} Oct 29 23:07:53.098292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.098263 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" event={"ID":"42ddbbdc-958f-44f6-bf08-aad1877e8875","Type":"ContainerDied","Data":"d42139e54723146b6436fb6f6bf844670f097110ea3a7a67d0c1bc36b81fb8b4"} Oct 29 23:07:53.098410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.098293 5736 scope.go:122] "RemoveContainer" containerID="db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92" Oct 29 23:07:53.098468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.098450 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk" Oct 29 23:07:53.104248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.104218 5736 generic.go:356] "Generic (PLEG): container finished" podID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerID="608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c" exitCode=0 Oct 29 23:07:53.104349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.104329 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" event={"ID":"024e7072-b667-4a09-ab8e-6eadbce1c464","Type":"ContainerDied","Data":"608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c"} Oct 29 23:07:53.104392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.104381 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" event={"ID":"024e7072-b667-4a09-ab8e-6eadbce1c464","Type":"ContainerDied","Data":"b8d40ba52a24c99d4b2e2e17ea757367625f68e94e37ecf2803583e2b80be714"} Oct 29 23:07:53.104474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.104462 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth" Oct 29 23:07:53.107313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.107285 5736 generic.go:356] "Generic (PLEG): container finished" podID="d84035ea-6637-46bb-915c-7912e9734d70" containerID="736421b4c3fc58fa47af1147a92388e1fbfc2b2cd19655bf046c1f79d5b4703e" exitCode=0 Oct 29 23:07:53.107469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.107451 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" event={"ID":"d84035ea-6637-46bb-915c-7912e9734d70","Type":"ContainerDied","Data":"736421b4c3fc58fa47af1147a92388e1fbfc2b2cd19655bf046c1f79d5b4703e"} Oct 29 23:07:53.118436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.118413 5736 scope.go:122] "RemoveContainer" containerID="db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92" Oct 29 23:07:53.118814 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:07:53.118785 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92\": container with ID starting with db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92 not found: ID does not exist" containerID="db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92" Oct 29 23:07:53.118922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.118826 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92"} err="failed to get container status \"db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92\": rpc error: code = NotFound desc = could not find container \"db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92\": container with ID starting with db8eddc38bce682c455413935fa2647bf511faeee5a5c2b6587fa4e9cf4d4a92 not found: ID does not exist" Oct 29 23:07:53.118922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.118852 5736 scope.go:122] "RemoveContainer" containerID="608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c" Oct 29 23:07:53.184476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.151744 5736 scope.go:122] "RemoveContainer" containerID="608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c" Oct 29 23:07:53.184476 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:07:53.152105 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c\": container with ID starting with 608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c not found: ID does not exist" containerID="608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c" Oct 29 23:07:53.184476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.152142 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c"} err="failed to get container status \"608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c\": rpc error: code = NotFound desc = could not find container \"608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c\": container with ID starting with 608e7465148cac49602af207e9f5e3e7dd9f9cc69b76cad64f2ae91183c0072c not found: ID does not exist" Oct 29 23:07:53.184476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.172509 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk"] Oct 29 23:07:53.184476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.176848 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-66tfk"] Oct 29 23:07:53.218547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.218511 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth"] Oct 29 23:07:53.223505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.223475 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-w7kth"] Oct 29 23:07:53.305520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.305447 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.383980 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-envoy\") pod \"d84035ea-6637-46bb-915c-7912e9734d70\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.384066 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-socket\") pod \"d84035ea-6637-46bb-915c-7912e9734d70\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.384150 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-kube-api-access-s5kt8\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-kube-api-access-s5kt8\") pod \"d84035ea-6637-46bb-915c-7912e9734d70\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.384230 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d84035ea-6637-46bb-915c-7912e9734d70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84035ea-6637-46bb-915c-7912e9734d70-istio-podinfo\") pod \"d84035ea-6637-46bb-915c-7912e9734d70\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.384328 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-certs\") pod \"d84035ea-6637-46bb-915c-7912e9734d70\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.384392 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-data\") pod \"d84035ea-6637-46bb-915c-7912e9734d70\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.384606 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-istio-token\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-istio-token\") pod \"d84035ea-6637-46bb-915c-7912e9734d70\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.384672 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-credential-socket\") pod \"d84035ea-6637-46bb-915c-7912e9734d70\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.384742 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d84035ea-6637-46bb-915c-7912e9734d70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84035ea-6637-46bb-915c-7912e9734d70-istiod-ca-cert\") pod \"d84035ea-6637-46bb-915c-7912e9734d70\" (UID: \"d84035ea-6637-46bb-915c-7912e9734d70\") " Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.385940 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84035ea-6637-46bb-915c-7912e9734d70-istiod-ca-cert" pod "d84035ea-6637-46bb-915c-7912e9734d70" (UID: "d84035ea-6637-46bb-915c-7912e9734d70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.386254 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-certs" pod "d84035ea-6637-46bb-915c-7912e9734d70" (UID: "d84035ea-6637-46bb-915c-7912e9734d70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.386446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.386438 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-data" pod "d84035ea-6637-46bb-915c-7912e9734d70" (UID: "d84035ea-6637-46bb-915c-7912e9734d70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.391325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.389238 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-istio-token" pod "d84035ea-6637-46bb-915c-7912e9734d70" (UID: "d84035ea-6637-46bb-915c-7912e9734d70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:53.391325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.389435 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-credential-socket" pod "d84035ea-6637-46bb-915c-7912e9734d70" (UID: "d84035ea-6637-46bb-915c-7912e9734d70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.391507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.391483 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-socket" pod "d84035ea-6637-46bb-915c-7912e9734d70" (UID: "d84035ea-6637-46bb-915c-7912e9734d70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.394500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.393163 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-envoy" pod "d84035ea-6637-46bb-915c-7912e9734d70" (UID: "d84035ea-6637-46bb-915c-7912e9734d70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:53.394500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.393692 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-kube-api-access-s5kt8" pod "d84035ea-6637-46bb-915c-7912e9734d70" (UID: "d84035ea-6637-46bb-915c-7912e9734d70"). InnerVolumeSpecName "kube-api-access-s5kt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:53.395618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.394988 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d84035ea-6637-46bb-915c-7912e9734d70-istio-podinfo" pod "d84035ea-6637-46bb-915c-7912e9734d70" (UID: "d84035ea-6637-46bb-915c-7912e9734d70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:53.487596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.487550 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84035ea-6637-46bb-915c-7912e9734d70-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.487596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.487602 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.487797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.487619 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.487797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.487638 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5kt8\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-kube-api-access-s5kt8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.487797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.487653 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84035ea-6637-46bb-915c-7912e9734d70-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.487797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.487667 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.487797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.487680 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.487797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.487692 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84035ea-6637-46bb-915c-7912e9734d70-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:53.487797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:53.487705 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84035ea-6637-46bb-915c-7912e9734d70-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:07:54.118111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:54.118079 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" Oct 29 23:07:54.118111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:54.118102 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn" event={"ID":"d84035ea-6637-46bb-915c-7912e9734d70","Type":"ContainerDied","Data":"489b97dfdd9565c642579b2e41ce280ebdb34d5eff88074c6de9c1598f2528cf"} Oct 29 23:07:54.118624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:54.118167 5736 scope.go:122] "RemoveContainer" containerID="736421b4c3fc58fa47af1147a92388e1fbfc2b2cd19655bf046c1f79d5b4703e" Oct 29 23:07:54.191825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:54.191784 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn"] Oct 29 23:07:54.203573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:54.203541 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-fv7dn"] Oct 29 23:07:55.004447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:55.004415 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" path="/var/lib/kubelet/pods/024e7072-b667-4a09-ab8e-6eadbce1c464/volumes" Oct 29 23:07:55.005057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:55.005040 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" path="/var/lib/kubelet/pods/42ddbbdc-958f-44f6-bf08-aad1877e8875/volumes" Oct 29 23:07:55.005505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:07:55.005491 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84035ea-6637-46bb-915c-7912e9734d70" path="/var/lib/kubelet/pods/d84035ea-6637-46bb-915c-7912e9734d70/volumes" Oct 29 23:07:56.824384 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:07:56.824347 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:08:00.205051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:00.204970 5736 generic.go:356] "Generic (PLEG): container finished" podID="9569f274-5c9b-4a99-81f2-7321183979b2" containerID="045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb" exitCode=0 Oct 29 23:08:00.205051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:00.205027 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" event={"ID":"9569f274-5c9b-4a99-81f2-7321183979b2","Type":"ContainerDied","Data":"045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb"} Oct 29 23:08:00.205611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:00.205418 5736 scope.go:122] "RemoveContainer" containerID="045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb" Oct 29 23:08:00.206995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:00.206966 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:00.207053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:00.207037 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:00.207090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:00.207069 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:01.213634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:01.213563 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" event={"ID":"9569f274-5c9b-4a99-81f2-7321183979b2","Type":"ContainerStarted","Data":"6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab"} Oct 29 23:08:01.252868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:01.252838 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:08:01.253090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:01.253068 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:08:01.253153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:01.253114 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 23:08:02.256199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:02.256112 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:08:02.256856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:02.256206 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 23:08:02.363504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:02.363469 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p"] Oct 29 23:08:02.375749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:02.375715 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0941c8-p9v8p"] Oct 29 23:08:03.006775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:03.006743 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2871a25b-e0ad-4b03-ac69-fb11942c43c5" path="/var/lib/kubelet/pods/2871a25b-e0ad-4b03-ac69-fb11942c43c5/volumes" Oct 29 23:08:03.255044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:03.255008 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:08:03.255341 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:03.255314 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 23:08:04.310789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:04.310751 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:08:04.311384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:04.311350 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 23:08:05.256009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:05.255968 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:08:05.256224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:05.256043 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 23:08:06.256044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:06.255966 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 23:08:06.256044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:06.256025 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 23:08:07.297866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:07.297828 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:08:07.298925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:07.298894 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:08:07.300426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:07.300400 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:08:10.148354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.148317 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn"] Oct 29 23:08:10.149117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149090 5736 cpu_manager.go:414] "containerMap: removing container" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerName="istio-proxy" Oct 29 23:08:10.149117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149116 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerName="istio-proxy" Oct 29 23:08:10.149339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149156 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" Oct 29 23:08:10.149339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149165 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" Oct 29 23:08:10.149339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149185 5736 cpu_manager.go:414] "containerMap: removing container" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" Oct 29 23:08:10.149339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149193 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" Oct 29 23:08:10.149339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149235 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3b62de84-4b61-44e4-801f-48f3c7b7fbb4" containerName="revoke-keys" Oct 29 23:08:10.149339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149242 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b62de84-4b61-44e4-801f-48f3c7b7fbb4" containerName="revoke-keys" Oct 29 23:08:10.149558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149371 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="024e7072-b667-4a09-ab8e-6eadbce1c464" containerName="istio-proxy" Oct 29 23:08:10.149558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149388 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b62de84-4b61-44e4-801f-48f3c7b7fbb4" containerName="revoke-keys" Oct 29 23:08:10.149558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149398 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ddbbdc-958f-44f6-bf08-aad1877e8875" containerName="istio-proxy" Oct 29 23:08:10.149558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.149407 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84035ea-6637-46bb-915c-7912e9734d70" containerName="istio-proxy" Oct 29 23:08:10.153208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.153188 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.168797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.168768 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:10.172770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.172747 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn"] Oct 29 23:08:10.180958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.180938 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:08:10.258495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.258464 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.258684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.258668 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.258826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.258804 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtj2r\" (UniqueName: \"kubernetes.io/projected/e07d200c-59c7-463b-af25-e6f4622f2294-kube-api-access-dtj2r\") pod \"maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.359696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.359663 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtj2r\" (UniqueName: \"kubernetes.io/projected/e07d200c-59c7-463b-af25-e6f4622f2294-kube-api-access-dtj2r\") pod \"maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.359860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.359750 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.359939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.359923 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.360422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.360400 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.362187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.362163 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.382984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.382955 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtj2r\" (UniqueName: \"kubernetes.io/projected/e07d200c-59c7-463b-af25-e6f4622f2294-kube-api-access-dtj2r\") pod \"maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.474642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.473376 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:10.911641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.911552 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn"] Oct 29 23:08:10.915850 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:08:10.915819 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07d200c_59c7_463b_af25_e6f4622f2294.slice/crio-74e732e3a257cb59fb3320f130982b29044b0302b0570228a2ee0193b93adc98 WatchSource:0}: Error finding container 74e732e3a257cb59fb3320f130982b29044b0302b0570228a2ee0193b93adc98: Status 404 returned error can't find the container with id 74e732e3a257cb59fb3320f130982b29044b0302b0570228a2ee0193b93adc98 Oct 29 23:08:10.956464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.954227 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9"] Oct 29 23:08:10.960439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.960416 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:10.994524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:10.991286 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9"] Oct 29 23:08:11.082188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.082143 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h845z\" (UniqueName: \"kubernetes.io/projected/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-kube-api-access-h845z\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.082344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.082323 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.082663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.082629 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.183623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.183506 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h845z\" (UniqueName: \"kubernetes.io/projected/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-kube-api-access-h845z\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.184074 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.183646 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.184074 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.183837 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.184444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.184423 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.186054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.186030 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.207093 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.207069 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h845z\" (UniqueName: \"kubernetes.io/projected/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-kube-api-access-h845z\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.292500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.292478 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:11.424314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.424272 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" event={"ID":"e07d200c-59c7-463b-af25-e6f4622f2294","Type":"ContainerStarted","Data":"04af90b93b20720b203418957724b4cb572974f62df6d684f1d540aecc5179ae"} Oct 29 23:08:11.424475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.424344 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" event={"ID":"e07d200c-59c7-463b-af25-e6f4622f2294","Type":"ContainerStarted","Data":"74e732e3a257cb59fb3320f130982b29044b0302b0570228a2ee0193b93adc98"} Oct 29 23:08:11.468647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.468598 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" podStartSLOduration=1.468565396 podStartE2EDuration="1.468565396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:11.465362815 +0000 UTC m=+1488.581282421" watchObservedRunningTime="2026-10-29 23:08:11.468565396 +0000 UTC m=+1488.584485001" Oct 29 23:08:11.617971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:11.617923 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9"] Oct 29 23:08:11.620673 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:08:11.620634 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6382bf4e_13ba_43d9_a75b_a3efa27c5cb3.slice/crio-40e5f174571039df1711604014e027eb113525eab16e4fc014754837457c2521 WatchSource:0}: Error finding container 40e5f174571039df1711604014e027eb113525eab16e4fc014754837457c2521: Status 404 returned error can't find the container with id 40e5f174571039df1711604014e027eb113525eab16e4fc014754837457c2521 Oct 29 23:08:12.433567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:12.433525 5736 generic.go:356] "Generic (PLEG): container finished" podID="e07d200c-59c7-463b-af25-e6f4622f2294" containerID="04af90b93b20720b203418957724b4cb572974f62df6d684f1d540aecc5179ae" exitCode=0 Oct 29 23:08:12.434080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:12.433604 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" event={"ID":"e07d200c-59c7-463b-af25-e6f4622f2294","Type":"ContainerDied","Data":"04af90b93b20720b203418957724b4cb572974f62df6d684f1d540aecc5179ae"} Oct 29 23:08:12.435618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:12.435573 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" event={"ID":"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3","Type":"ContainerStarted","Data":"6220bcf8593d148d3b180518dc3b91367df21e902e29919a648e3f7f2f4f96a5"} Oct 29 23:08:12.435734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:12.435636 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" event={"ID":"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3","Type":"ContainerStarted","Data":"40e5f174571039df1711604014e027eb113525eab16e4fc014754837457c2521"} Oct 29 23:08:12.524828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:12.524772 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" podStartSLOduration=2.5247592389999998 podStartE2EDuration="2.524759239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:12.520658374 +0000 UTC m=+1489.636577979" watchObservedRunningTime="2026-10-29 23:08:12.524759239 +0000 UTC m=+1489.640678844" Oct 29 23:08:13.444403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.444368 5736 generic.go:356] "Generic (PLEG): container finished" podID="6382bf4e-13ba-43d9-a75b-a3efa27c5cb3" containerID="6220bcf8593d148d3b180518dc3b91367df21e902e29919a648e3f7f2f4f96a5" exitCode=0 Oct 29 23:08:13.444826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.444457 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" event={"ID":"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3","Type":"ContainerDied","Data":"6220bcf8593d148d3b180518dc3b91367df21e902e29919a648e3f7f2f4f96a5"} Oct 29 23:08:13.597077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.597050 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:13.719436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.719405 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07d200c-59c7-463b-af25-e6f4622f2294-kube-api-access-dtj2r\" (UniqueName: \"kubernetes.io/projected/e07d200c-59c7-463b-af25-e6f4622f2294-kube-api-access-dtj2r\") pod \"e07d200c-59c7-463b-af25-e6f4622f2294\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " Oct 29 23:08:13.719608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.719454 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-xks-ca\") pod \"e07d200c-59c7-463b-af25-e6f4622f2294\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " Oct 29 23:08:13.719608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.719567 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-openshift-ca\") pod \"e07d200c-59c7-463b-af25-e6f4622f2294\" (UID: \"e07d200c-59c7-463b-af25-e6f4622f2294\") " Oct 29 23:08:13.719914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.719884 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-openshift-ca" pod "e07d200c-59c7-463b-af25-e6f4622f2294" (UID: "e07d200c-59c7-463b-af25-e6f4622f2294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:13.720268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.720247 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:13.721630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.721607 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-xks-ca" pod "e07d200c-59c7-463b-af25-e6f4622f2294" (UID: "e07d200c-59c7-463b-af25-e6f4622f2294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:13.721845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.721817 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07d200c-59c7-463b-af25-e6f4622f2294-kube-api-access-dtj2r" pod "e07d200c-59c7-463b-af25-e6f4622f2294" (UID: "e07d200c-59c7-463b-af25-e6f4622f2294"). InnerVolumeSpecName "kube-api-access-dtj2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:13.822008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.821975 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtj2r\" (UniqueName: \"kubernetes.io/projected/e07d200c-59c7-463b-af25-e6f4622f2294-kube-api-access-dtj2r\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:13.822008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:13.822003 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07d200c-59c7-463b-af25-e6f4622f2294-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:14.452308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.452283 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" Oct 29 23:08:14.452855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.452327 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn" event={"ID":"e07d200c-59c7-463b-af25-e6f4622f2294","Type":"ContainerDied","Data":"74e732e3a257cb59fb3320f130982b29044b0302b0570228a2ee0193b93adc98"} Oct 29 23:08:14.452855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.452376 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e732e3a257cb59fb3320f130982b29044b0302b0570228a2ee0193b93adc98" Oct 29 23:08:14.635312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.635282 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:14.733492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.733467 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-kube-api-access-h845z\" (UniqueName: \"kubernetes.io/projected/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-kube-api-access-h845z\") pod \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " Oct 29 23:08:14.733717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.733520 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-openshift-ca\") pod \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " Oct 29 23:08:14.733717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.733641 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-xks-ca\") pod \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\" (UID: \"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3\") " Oct 29 23:08:14.733979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.733955 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-openshift-ca" pod "6382bf4e-13ba-43d9-a75b-a3efa27c5cb3" (UID: "6382bf4e-13ba-43d9-a75b-a3efa27c5cb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:14.734254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.734236 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:14.735620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.735557 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-xks-ca" pod "6382bf4e-13ba-43d9-a75b-a3efa27c5cb3" (UID: "6382bf4e-13ba-43d9-a75b-a3efa27c5cb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:14.735716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.735620 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-kube-api-access-h845z" pod "6382bf4e-13ba-43d9-a75b-a3efa27c5cb3" (UID: "6382bf4e-13ba-43d9-a75b-a3efa27c5cb3"). InnerVolumeSpecName "kube-api-access-h845z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:14.836143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.836109 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h845z\" (UniqueName: \"kubernetes.io/projected/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-kube-api-access-h845z\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:14.836143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:14.836137 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:15.462327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.462298 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" Oct 29 23:08:15.462327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.462300 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9" event={"ID":"6382bf4e-13ba-43d9-a75b-a3efa27c5cb3","Type":"ContainerDied","Data":"40e5f174571039df1711604014e027eb113525eab16e4fc014754837457c2521"} Oct 29 23:08:15.462746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.462383 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40e5f174571039df1711604014e027eb113525eab16e4fc014754837457c2521" Oct 29 23:08:15.978706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.978668 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf"] Oct 29 23:08:15.979388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.979363 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6382bf4e-13ba-43d9-a75b-a3efa27c5cb3" containerName="revoke-keys" Oct 29 23:08:15.979499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.979389 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6382bf4e-13ba-43d9-a75b-a3efa27c5cb3" containerName="revoke-keys" Oct 29 23:08:15.979499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.979465 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e07d200c-59c7-463b-af25-e6f4622f2294" containerName="revoke-keys" Oct 29 23:08:15.979499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.979475 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07d200c-59c7-463b-af25-e6f4622f2294" containerName="revoke-keys" Oct 29 23:08:15.979664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.979640 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6382bf4e-13ba-43d9-a75b-a3efa27c5cb3" containerName="revoke-keys" Oct 29 23:08:15.979708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.979671 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07d200c-59c7-463b-af25-e6f4622f2294" containerName="revoke-keys" Oct 29 23:08:15.983448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.983431 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:15.999617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:15.999568 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7af1a5-openshift-default-dockercfg-5hnvl\"" Oct 29 23:08:16.014995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.014973 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf"] Oct 29 23:08:16.050922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.050884 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-token\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.051064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.050953 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-podinfo\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.051107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.051073 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-data\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.051167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.051124 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-credential-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.051239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.051221 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.051326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.051310 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-certs\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.051390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.051369 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm54s\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-kube-api-access-bm54s\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.051438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.051424 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a44adc-0ef9-45aa-a229-1857e9617cad-istiod-ca-cert\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.051474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.051463 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-envoy\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.152433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.152399 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm54s\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-kube-api-access-bm54s\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.152616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.152445 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a44adc-0ef9-45aa-a229-1857e9617cad-istiod-ca-cert\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.152616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.152481 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-envoy\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.152616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.152519 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-token\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.152616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.152563 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-podinfo\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.152862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.152707 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-data\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.152862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.152777 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-credential-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.152862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.152818 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.153035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.152943 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-certs\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.153123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.153091 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-data\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.153200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.153178 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-credential-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.153301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.153278 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-certs\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.153368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.153313 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-socket\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.153368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.153358 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a44adc-0ef9-45aa-a229-1857e9617cad-istiod-ca-cert\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.154980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.154956 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-envoy\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.155188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.155170 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-podinfo\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.175861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.175833 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-token\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.175973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.175958 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm54s\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-kube-api-access-bm54s\") pod \"e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.297475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.297391 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:16.630445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.630416 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf"] Oct 29 23:08:16.633534 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:08:16.633499 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a44adc_0ef9_45aa_a229_1857e9617cad.slice/crio-d8b95e303976899999844decc30fd4f325236b4b6440e1aacfef5babf3d89e61 WatchSource:0}: Error finding container d8b95e303976899999844decc30fd4f325236b4b6440e1aacfef5babf3d89e61: Status 404 returned error can't find the container with id d8b95e303976899999844decc30fd4f325236b4b6440e1aacfef5babf3d89e61 Oct 29 23:08:16.636319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.636287 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:16.636413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.636361 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:16.636413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:16.636395 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:17.476047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.476012 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv"] Oct 29 23:08:17.481230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.481207 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.481382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.481281 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" event={"ID":"34a44adc-0ef9-45aa-a229-1857e9617cad","Type":"ContainerStarted","Data":"cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c"} Oct 29 23:08:17.481382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.481344 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" event={"ID":"34a44adc-0ef9-45aa-a229-1857e9617cad","Type":"ContainerStarted","Data":"d8b95e303976899999844decc30fd4f325236b4b6440e1aacfef5babf3d89e61"} Oct 29 23:08:17.498016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.497996 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:17.510375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.510313 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:08:17.516112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.516084 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv"] Oct 29 23:08:17.574913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.574850 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" podStartSLOduration=2.574822912 podStartE2EDuration="2.574822912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:17.571391008 +0000 UTC m=+1494.687310615" watchObservedRunningTime="2026-10-29 23:08:17.574822912 +0000 UTC m=+1494.690742519" Oct 29 23:08:17.587232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.587202 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f26nc\" (UniqueName: \"kubernetes.io/projected/ef020a79-da27-4329-9396-da946adc90dc-kube-api-access-f26nc\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.587452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.587428 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef020a79-da27-4329-9396-da946adc90dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.587613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.587572 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef020a79-da27-4329-9396-da946adc90dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.689170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.689118 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f26nc\" (UniqueName: \"kubernetes.io/projected/ef020a79-da27-4329-9396-da946adc90dc-kube-api-access-f26nc\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.689570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.689273 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef020a79-da27-4329-9396-da946adc90dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.689570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.689342 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef020a79-da27-4329-9396-da946adc90dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.690098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.690071 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef020a79-da27-4329-9396-da946adc90dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.691805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.691776 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef020a79-da27-4329-9396-da946adc90dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.712137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.712113 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f26nc\" (UniqueName: \"kubernetes.io/projected/ef020a79-da27-4329-9396-da946adc90dc-kube-api-access-f26nc\") pod \"maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:17.795935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:17.795846 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:18.122616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:18.122593 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv"] Oct 29 23:08:18.124832 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:08:18.124802 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef020a79_da27_4329_9396_da946adc90dc.slice/crio-16a3661b123a64dc6c3416ce9063966677924be1513f7e1a50af06f22f7a7b04 WatchSource:0}: Error finding container 16a3661b123a64dc6c3416ce9063966677924be1513f7e1a50af06f22f7a7b04: Status 404 returned error can't find the container with id 16a3661b123a64dc6c3416ce9063966677924be1513f7e1a50af06f22f7a7b04 Oct 29 23:08:18.297968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:18.297935 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:18.299431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:18.299408 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:08:18.299563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:18.299451 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 23:08:18.493361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:18.493321 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" event={"ID":"ef020a79-da27-4329-9396-da946adc90dc","Type":"ContainerStarted","Data":"16a3661b123a64dc6c3416ce9063966677924be1513f7e1a50af06f22f7a7b04"} Oct 29 23:08:19.297859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:19.297823 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:08:19.298256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:19.297874 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 23:08:19.505817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:19.505785 5736 generic.go:356] "Generic (PLEG): container finished" podID="ef020a79-da27-4329-9396-da946adc90dc" containerID="378e6cfc2d8150ebbbe0eb3fba1edf1195c92829ea751d45286f6786113bee88" exitCode=0 Oct 29 23:08:19.505997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:19.505870 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" event={"ID":"ef020a79-da27-4329-9396-da946adc90dc","Type":"ContainerDied","Data":"378e6cfc2d8150ebbbe0eb3fba1edf1195c92829ea751d45286f6786113bee88"} Oct 29 23:08:20.297935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.297899 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:08:20.298393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.297966 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 23:08:20.822097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.821792 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:20.985330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.985292 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef020a79-da27-4329-9396-da946adc90dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef020a79-da27-4329-9396-da946adc90dc-maas-api-openshift-ca\") pod \"ef020a79-da27-4329-9396-da946adc90dc\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " Oct 29 23:08:20.985541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.985345 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef020a79-da27-4329-9396-da946adc90dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef020a79-da27-4329-9396-da946adc90dc-maas-api-xks-ca\") pod \"ef020a79-da27-4329-9396-da946adc90dc\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " Oct 29 23:08:20.985642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.985536 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef020a79-da27-4329-9396-da946adc90dc-kube-api-access-f26nc\" (UniqueName: \"kubernetes.io/projected/ef020a79-da27-4329-9396-da946adc90dc-kube-api-access-f26nc\") pod \"ef020a79-da27-4329-9396-da946adc90dc\" (UID: \"ef020a79-da27-4329-9396-da946adc90dc\") " Oct 29 23:08:20.986106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.986074 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef020a79-da27-4329-9396-da946adc90dc-maas-api-openshift-ca" pod "ef020a79-da27-4329-9396-da946adc90dc" (UID: "ef020a79-da27-4329-9396-da946adc90dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:20.986335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.986312 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef020a79-da27-4329-9396-da946adc90dc-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:20.988553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.988526 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef020a79-da27-4329-9396-da946adc90dc-maas-api-xks-ca" pod "ef020a79-da27-4329-9396-da946adc90dc" (UID: "ef020a79-da27-4329-9396-da946adc90dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:20.988991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:20.988966 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef020a79-da27-4329-9396-da946adc90dc-kube-api-access-f26nc" pod "ef020a79-da27-4329-9396-da946adc90dc" (UID: "ef020a79-da27-4329-9396-da946adc90dc"). InnerVolumeSpecName "kube-api-access-f26nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:21.087616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:21.087415 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef020a79-da27-4329-9396-da946adc90dc-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:21.087616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:21.087453 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f26nc\" (UniqueName: \"kubernetes.io/projected/ef020a79-da27-4329-9396-da946adc90dc-kube-api-access-f26nc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:21.308981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:21.308477 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:08:21.308981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:21.308538 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 23:08:21.608405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:21.608309 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" event={"ID":"ef020a79-da27-4329-9396-da946adc90dc","Type":"ContainerDied","Data":"16a3661b123a64dc6c3416ce9063966677924be1513f7e1a50af06f22f7a7b04"} Oct 29 23:08:21.608781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:21.608670 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16a3661b123a64dc6c3416ce9063966677924be1513f7e1a50af06f22f7a7b04" Oct 29 23:08:21.609039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:21.608940 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv" Oct 29 23:08:22.298296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:22.298259 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 23:08:22.298476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:22.298310 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 23:08:23.304199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:23.304169 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:23.305125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:23.305100 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:23.306046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:23.306025 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:08:23.336736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:23.336711 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:08:23.338222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:23.338183 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:08:24.367969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:24.367929 5736 scope.go:122] "RemoveContainer" containerID="f3852526518086a0d93387d38ec6c2d48b8a0d7822fbaf4f1ebe7ad7c8ab1a41" Oct 29 23:08:24.378246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:24.378220 5736 scope.go:122] "RemoveContainer" containerID="cb5b91bd20fc4e836bd59b9ab519960a818911921b7c652337db8c5e5e4a7963" Oct 29 23:08:26.838689 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:08:26.834570 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:08:31.757152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:31.757110 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9"] Oct 29 23:08:31.779655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:31.779297 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5978-rq7h9"] Oct 29 23:08:33.006269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:33.006225 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6382bf4e-13ba-43d9-a75b-a3efa27c5cb3" path="/var/lib/kubelet/pods/6382bf4e-13ba-43d9-a75b-a3efa27c5cb3/volumes" Oct 29 23:08:34.663704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:34.663670 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv"] Oct 29 23:08:34.675727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:34.675697 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-dfedfe-7wzvv"] Oct 29 23:08:35.004570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:35.004535 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef020a79-da27-4329-9396-da946adc90dc" path="/var/lib/kubelet/pods/ef020a79-da27-4329-9396-da946adc90dc/volumes" Oct 29 23:08:36.839147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:36.839110 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch"] Oct 29 23:08:36.839547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:36.839392 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" containerID="cri-o://4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460" gracePeriod=30 Oct 29 23:08:40.152225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.152186 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:08:40.152716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.152245 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:08:40.247770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.247740 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d"] Oct 29 23:08:40.248360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.248342 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ef020a79-da27-4329-9396-da946adc90dc" containerName="revoke-keys" Oct 29 23:08:40.248360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.248359 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef020a79-da27-4329-9396-da946adc90dc" containerName="revoke-keys" Oct 29 23:08:40.248501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.248467 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef020a79-da27-4329-9396-da946adc90dc" containerName="revoke-keys" Oct 29 23:08:40.254011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.253992 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.275525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.275501 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:40.284555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.284525 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvgbp\" (UniqueName: \"kubernetes.io/projected/adff96c3-569c-409a-8b5f-996e97e797d3-kube-api-access-zvgbp\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.284701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.284684 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.284818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.284802 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.287910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.287882 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:08:40.299845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.299815 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d"] Oct 29 23:08:40.386369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.386339 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgbp\" (UniqueName: \"kubernetes.io/projected/adff96c3-569c-409a-8b5f-996e97e797d3-kube-api-access-zvgbp\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.386504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.386408 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.386504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.386466 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.387039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.387011 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.388760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.388739 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.409336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.409281 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgbp\" (UniqueName: \"kubernetes.io/projected/adff96c3-569c-409a-8b5f-996e97e797d3-kube-api-access-zvgbp\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.567434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.567399 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:40.651735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.649434 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd"] Oct 29 23:08:40.651735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.649702 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" containerID="cri-o://6682af431b576bf67a325b810713479738eaf4a1cb50525db42496a12011b2a8" gracePeriod=30 Oct 29 23:08:40.881056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:40.881030 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d"] Oct 29 23:08:40.884235 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:08:40.884206 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadff96c3_569c_409a_8b5f_996e97e797d3.slice/crio-deb6708c9e50a6e556ea5108c5a43c709f0a83b3a8562a58993d70e1e9759c7b WatchSource:0}: Error finding container deb6708c9e50a6e556ea5108c5a43c709f0a83b3a8562a58993d70e1e9759c7b: Status 404 returned error can't find the container with id deb6708c9e50a6e556ea5108c5a43c709f0a83b3a8562a58993d70e1e9759c7b Oct 29 23:08:41.773442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:41.773399 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" event={"ID":"adff96c3-569c-409a-8b5f-996e97e797d3","Type":"ContainerStarted","Data":"40ee5546b6aaf1347b61ad309a9aaedb6deb05ffef8c93aa44aac5c423504bd6"} Oct 29 23:08:41.773953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:41.773470 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" event={"ID":"adff96c3-569c-409a-8b5f-996e97e797d3","Type":"ContainerStarted","Data":"deb6708c9e50a6e556ea5108c5a43c709f0a83b3a8562a58993d70e1e9759c7b"} Oct 29 23:08:42.203607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.199470 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 23:08:42.339433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.339337 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54cfaf69-db9f-4bee-ac82-b012b725172d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cfaf69-db9f-4bee-ac82-b012b725172d-istiod-ca-cert\") pod \"54cfaf69-db9f-4bee-ac82-b012b725172d\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " Oct 29 23:08:42.339433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.339376 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-podinfo\") pod \"54cfaf69-db9f-4bee-ac82-b012b725172d\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " Oct 29 23:08:42.339433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.339406 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-token\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-token\") pod \"54cfaf69-db9f-4bee-ac82-b012b725172d\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " Oct 29 23:08:42.339774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.339500 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-envoy\") pod \"54cfaf69-db9f-4bee-ac82-b012b725172d\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " Oct 29 23:08:42.339774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.339631 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-data\") pod \"54cfaf69-db9f-4bee-ac82-b012b725172d\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " Oct 29 23:08:42.339774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.339701 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cfaf69-db9f-4bee-ac82-b012b725172d-istiod-ca-cert" pod "54cfaf69-db9f-4bee-ac82-b012b725172d" (UID: "54cfaf69-db9f-4bee-ac82-b012b725172d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:42.339933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.339887 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-data" pod "54cfaf69-db9f-4bee-ac82-b012b725172d" (UID: "54cfaf69-db9f-4bee-ac82-b012b725172d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.340016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.339998 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-kube-api-access-fkmnn\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-kube-api-access-fkmnn\") pod \"54cfaf69-db9f-4bee-ac82-b012b725172d\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " Oct 29 23:08:42.340098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.340081 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-socket\") pod \"54cfaf69-db9f-4bee-ac82-b012b725172d\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " Oct 29 23:08:42.340620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.340336 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-socket" pod "54cfaf69-db9f-4bee-ac82-b012b725172d" (UID: "54cfaf69-db9f-4bee-ac82-b012b725172d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.340751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.340731 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-certs\") pod \"54cfaf69-db9f-4bee-ac82-b012b725172d\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " Oct 29 23:08:42.340831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.340792 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-credential-socket\") pod \"54cfaf69-db9f-4bee-ac82-b012b725172d\" (UID: \"54cfaf69-db9f-4bee-ac82-b012b725172d\") " Oct 29 23:08:42.341058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.341012 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-certs" pod "54cfaf69-db9f-4bee-ac82-b012b725172d" (UID: "54cfaf69-db9f-4bee-ac82-b012b725172d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.341249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.341227 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-credential-socket" pod "54cfaf69-db9f-4bee-ac82-b012b725172d" (UID: "54cfaf69-db9f-4bee-ac82-b012b725172d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.341486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.341471 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.341565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.341487 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.341565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.341497 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.341565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.341505 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.341565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.341513 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cfaf69-db9f-4bee-ac82-b012b725172d-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.342137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.342118 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-token" pod "54cfaf69-db9f-4bee-ac82-b012b725172d" (UID: "54cfaf69-db9f-4bee-ac82-b012b725172d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:42.342219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.342154 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-envoy" pod "54cfaf69-db9f-4bee-ac82-b012b725172d" (UID: "54cfaf69-db9f-4bee-ac82-b012b725172d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:42.342219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.342165 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-podinfo" pod "54cfaf69-db9f-4bee-ac82-b012b725172d" (UID: "54cfaf69-db9f-4bee-ac82-b012b725172d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:42.342302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.342232 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-kube-api-access-fkmnn" pod "54cfaf69-db9f-4bee-ac82-b012b725172d" (UID: "54cfaf69-db9f-4bee-ac82-b012b725172d"). InnerVolumeSpecName "kube-api-access-fkmnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:42.443907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.443807 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkmnn\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-kube-api-access-fkmnn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.443907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.443847 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.443907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.443861 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.443907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.443875 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cfaf69-db9f-4bee-ac82-b012b725172d-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:42.625121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.625050 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv"] Oct 29 23:08:42.625699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.625682 5736 cpu_manager.go:414] "containerMap: removing container" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" Oct 29 23:08:42.625746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.625700 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" Oct 29 23:08:42.625836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.625825 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" Oct 29 23:08:42.629674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.629650 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.648630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.648559 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv"] Oct 29 23:08:42.747561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.747514 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.747801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.747752 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.747876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.747803 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clkp8\" (UniqueName: \"kubernetes.io/projected/88ecad2a-3820-48b3-ae06-ec40179ac8ca-kube-api-access-clkp8\") pod \"maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.783059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.783031 5736 generic.go:356] "Generic (PLEG): container finished" podID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerID="4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460" exitCode=0 Oct 29 23:08:42.783523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.783081 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" Oct 29 23:08:42.783523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.783118 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" event={"ID":"54cfaf69-db9f-4bee-ac82-b012b725172d","Type":"ContainerDied","Data":"4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460"} Oct 29 23:08:42.783523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.783189 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch" event={"ID":"54cfaf69-db9f-4bee-ac82-b012b725172d","Type":"ContainerDied","Data":"101f0d14a92d2634872a3f94c491ae0fbf511900fc66e6dfb4b27f32ffbdd072"} Oct 29 23:08:42.783523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.783237 5736 scope.go:122] "RemoveContainer" containerID="4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460" Oct 29 23:08:42.785425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.785403 5736 generic.go:356] "Generic (PLEG): container finished" podID="adff96c3-569c-409a-8b5f-996e97e797d3" containerID="40ee5546b6aaf1347b61ad309a9aaedb6deb05ffef8c93aa44aac5c423504bd6" exitCode=0 Oct 29 23:08:42.785569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.785545 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" event={"ID":"adff96c3-569c-409a-8b5f-996e97e797d3","Type":"ContainerDied","Data":"40ee5546b6aaf1347b61ad309a9aaedb6deb05ffef8c93aa44aac5c423504bd6"} Oct 29 23:08:42.796519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.796494 5736 scope.go:122] "RemoveContainer" containerID="421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06" Oct 29 23:08:42.808263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.808243 5736 scope.go:122] "RemoveContainer" containerID="4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460" Oct 29 23:08:42.808637 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:08:42.808573 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460\": container with ID starting with 4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460 not found: ID does not exist" containerID="4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460" Oct 29 23:08:42.808713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.808647 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460"} err="failed to get container status \"4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460\": rpc error: code = NotFound desc = could not find container \"4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460\": container with ID starting with 4fede410bee976c72408d606132071ff4f5b0011f976569e5b3097a4c29b9460 not found: ID does not exist" Oct 29 23:08:42.808713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.808666 5736 scope.go:122] "RemoveContainer" containerID="421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06" Oct 29 23:08:42.808975 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:08:42.808954 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06\": container with ID starting with 421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06 not found: ID does not exist" containerID="421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06" Oct 29 23:08:42.809022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.808980 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06"} err="failed to get container status \"421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06\": rpc error: code = NotFound desc = could not find container \"421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06\": container with ID starting with 421e0cc15feae203b7e4161e9c4cf852b474d740fae44c7f0f1da0721cabed06 not found: ID does not exist" Oct 29 23:08:42.837059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.837029 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch"] Oct 29 23:08:42.849185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.849152 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-xgwch"] Oct 29 23:08:42.850158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.850127 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.850298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.850176 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clkp8\" (UniqueName: \"kubernetes.io/projected/88ecad2a-3820-48b3-ae06-ec40179ac8ca-kube-api-access-clkp8\") pod \"maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.850419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.850398 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.851193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.851166 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.854630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.853355 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.874274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.874250 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clkp8\" (UniqueName: \"kubernetes.io/projected/88ecad2a-3820-48b3-ae06-ec40179ac8ca-kube-api-access-clkp8\") pod \"maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.944739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.944707 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:42.960955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:42.960931 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:43.011739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.011713 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" path="/var/lib/kubelet/pods/54cfaf69-db9f-4bee-ac82-b012b725172d/volumes" Oct 29 23:08:43.053144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.053102 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-openshift-ca\") pod \"adff96c3-569c-409a-8b5f-996e97e797d3\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " Oct 29 23:08:43.053378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.053269 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-xks-ca\") pod \"adff96c3-569c-409a-8b5f-996e97e797d3\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " Oct 29 23:08:43.053378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.053317 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adff96c3-569c-409a-8b5f-996e97e797d3-kube-api-access-zvgbp\" (UniqueName: \"kubernetes.io/projected/adff96c3-569c-409a-8b5f-996e97e797d3-kube-api-access-zvgbp\") pod \"adff96c3-569c-409a-8b5f-996e97e797d3\" (UID: \"adff96c3-569c-409a-8b5f-996e97e797d3\") " Oct 29 23:08:43.056088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.056060 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-openshift-ca" pod "adff96c3-569c-409a-8b5f-996e97e797d3" (UID: "adff96c3-569c-409a-8b5f-996e97e797d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:43.056370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.056351 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adff96c3-569c-409a-8b5f-996e97e797d3-kube-api-access-zvgbp" pod "adff96c3-569c-409a-8b5f-996e97e797d3" (UID: "adff96c3-569c-409a-8b5f-996e97e797d3"). InnerVolumeSpecName "kube-api-access-zvgbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:43.056731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.056698 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-xks-ca" pod "adff96c3-569c-409a-8b5f-996e97e797d3" (UID: "adff96c3-569c-409a-8b5f-996e97e797d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:43.155399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.155274 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.155399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.155308 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvgbp\" (UniqueName: \"kubernetes.io/projected/adff96c3-569c-409a-8b5f-996e97e797d3-kube-api-access-zvgbp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.155399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.155323 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adff96c3-569c-409a-8b5f-996e97e797d3-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.264871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.264844 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv"] Oct 29 23:08:43.266670 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:08:43.266640 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ecad2a_3820_48b3_ae06_ec40179ac8ca.slice/crio-caee6a9ee079cdd55fcdb278d331dcc5f31d423073939305b77af448fba5b896 WatchSource:0}: Error finding container caee6a9ee079cdd55fcdb278d331dcc5f31d423073939305b77af448fba5b896: Status 404 returned error can't find the container with id caee6a9ee079cdd55fcdb278d331dcc5f31d423073939305b77af448fba5b896 Oct 29 23:08:43.797319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.797234 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" Oct 29 23:08:43.797319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.797275 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d" event={"ID":"adff96c3-569c-409a-8b5f-996e97e797d3","Type":"ContainerDied","Data":"deb6708c9e50a6e556ea5108c5a43c709f0a83b3a8562a58993d70e1e9759c7b"} Oct 29 23:08:43.797876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.797357 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deb6708c9e50a6e556ea5108c5a43c709f0a83b3a8562a58993d70e1e9759c7b" Oct 29 23:08:43.800208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.800178 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" event={"ID":"88ecad2a-3820-48b3-ae06-ec40179ac8ca","Type":"ContainerStarted","Data":"30dfde2f9363822e5e47cce685506cb92fee8755173c6dae0290226d065c4c87"} Oct 29 23:08:43.800331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.800224 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" event={"ID":"88ecad2a-3820-48b3-ae06-ec40179ac8ca","Type":"ContainerStarted","Data":"caee6a9ee079cdd55fcdb278d331dcc5f31d423073939305b77af448fba5b896"} Oct 29 23:08:43.842480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:43.842429 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" podStartSLOduration=1.842415674 podStartE2EDuration="1.842415674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:43.840085648 +0000 UTC m=+1520.956005244" watchObservedRunningTime="2026-10-29 23:08:43.842415674 +0000 UTC m=+1520.958335279" Oct 29 23:08:44.809237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:44.809202 5736 generic.go:356] "Generic (PLEG): container finished" podID="88ecad2a-3820-48b3-ae06-ec40179ac8ca" containerID="30dfde2f9363822e5e47cce685506cb92fee8755173c6dae0290226d065c4c87" exitCode=0 Oct 29 23:08:44.809681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:44.809286 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" event={"ID":"88ecad2a-3820-48b3-ae06-ec40179ac8ca","Type":"ContainerDied","Data":"30dfde2f9363822e5e47cce685506cb92fee8755173c6dae0290226d065c4c87"} Oct 29 23:08:45.820431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:45.820398 5736 generic.go:356] "Generic (PLEG): container finished" podID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerID="6682af431b576bf67a325b810713479738eaf4a1cb50525db42496a12011b2a8" exitCode=0 Oct 29 23:08:45.820917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:45.820470 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" event={"ID":"44af3d7c-5be4-4f8c-a584-14f53dbfef70","Type":"ContainerDied","Data":"6682af431b576bf67a325b810713479738eaf4a1cb50525db42496a12011b2a8"} Oct 29 23:08:45.985959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:45.985927 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 23:08:46.031071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.031040 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:46.075436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.075404 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn"] Oct 29 23:08:46.080963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.080936 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-957eef-tcdzn"] Oct 29 23:08:46.142732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.142695 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-token\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-token\") pod \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " Oct 29 23:08:46.142900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.142808 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-openshift-ca\") pod \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " Oct 29 23:08:46.142900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.142835 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-socket\") pod \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " Oct 29 23:08:46.142900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.142870 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istiod-ca-cert\") pod \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " Oct 29 23:08:46.142900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.142891 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ecad2a-3820-48b3-ae06-ec40179ac8ca-kube-api-access-clkp8\" (UniqueName: \"kubernetes.io/projected/88ecad2a-3820-48b3-ae06-ec40179ac8ca-kube-api-access-clkp8\") pod \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " Oct 29 23:08:46.143132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.142915 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-credential-socket\") pod \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " Oct 29 23:08:46.143239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143210 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-openshift-ca" pod "88ecad2a-3820-48b3-ae06-ec40179ac8ca" (UID: "88ecad2a-3820-48b3-ae06-ec40179ac8ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:46.143311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143206 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-socket" pod "44af3d7c-5be4-4f8c-a584-14f53dbfef70" (UID: "44af3d7c-5be4-4f8c-a584-14f53dbfef70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:46.143381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143341 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-credential-socket" pod "44af3d7c-5be4-4f8c-a584-14f53dbfef70" (UID: "44af3d7c-5be4-4f8c-a584-14f53dbfef70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:46.143381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143365 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istiod-ca-cert" pod "44af3d7c-5be4-4f8c-a584-14f53dbfef70" (UID: "44af3d7c-5be4-4f8c-a584-14f53dbfef70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:46.143511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143493 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-xks-ca\") pod \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\" (UID: \"88ecad2a-3820-48b3-ae06-ec40179ac8ca\") " Oct 29 23:08:46.143564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143550 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-certs\") pod \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " Oct 29 23:08:46.143650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143634 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-envoy\") pod \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " Oct 29 23:08:46.143713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143668 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-data\") pod \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " Oct 29 23:08:46.143857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143834 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-certs" pod "44af3d7c-5be4-4f8c-a584-14f53dbfef70" (UID: "44af3d7c-5be4-4f8c-a584-14f53dbfef70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:46.143970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143878 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-kube-api-access-5z9dn\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-kube-api-access-5z9dn\") pod \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " Oct 29 23:08:46.144023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.143967 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-podinfo\") pod \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\" (UID: \"44af3d7c-5be4-4f8c-a584-14f53dbfef70\") " Oct 29 23:08:46.144128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.144108 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-data" pod "44af3d7c-5be4-4f8c-a584-14f53dbfef70" (UID: "44af3d7c-5be4-4f8c-a584-14f53dbfef70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:46.145021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.144991 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.145144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.145026 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.145144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.145041 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.145144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.145056 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.145144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.145070 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.145144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.145084 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.145453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.145386 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-token" pod "44af3d7c-5be4-4f8c-a584-14f53dbfef70" (UID: "44af3d7c-5be4-4f8c-a584-14f53dbfef70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:46.145774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.145743 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ecad2a-3820-48b3-ae06-ec40179ac8ca-kube-api-access-clkp8" pod "88ecad2a-3820-48b3-ae06-ec40179ac8ca" (UID: "88ecad2a-3820-48b3-ae06-ec40179ac8ca"). InnerVolumeSpecName "kube-api-access-clkp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:46.146164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.146135 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-envoy" pod "44af3d7c-5be4-4f8c-a584-14f53dbfef70" (UID: "44af3d7c-5be4-4f8c-a584-14f53dbfef70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:46.146282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.146234 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-xks-ca" pod "88ecad2a-3820-48b3-ae06-ec40179ac8ca" (UID: "88ecad2a-3820-48b3-ae06-ec40179ac8ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:46.146464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.146446 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-podinfo" pod "44af3d7c-5be4-4f8c-a584-14f53dbfef70" (UID: "44af3d7c-5be4-4f8c-a584-14f53dbfef70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:46.147045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.147021 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-kube-api-access-5z9dn" pod "44af3d7c-5be4-4f8c-a584-14f53dbfef70" (UID: "44af3d7c-5be4-4f8c-a584-14f53dbfef70"). InnerVolumeSpecName "kube-api-access-5z9dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:46.246319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.246277 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z9dn\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-kube-api-access-5z9dn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.246319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.246312 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.246319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.246322 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.246619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.246331 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clkp8\" (UniqueName: \"kubernetes.io/projected/88ecad2a-3820-48b3-ae06-ec40179ac8ca-kube-api-access-clkp8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.246619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.246340 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ecad2a-3820-48b3-ae06-ec40179ac8ca-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.246619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.246350 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44af3d7c-5be4-4f8c-a584-14f53dbfef70-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:46.828968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.828938 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" Oct 29 23:08:46.828968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.828953 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd" event={"ID":"44af3d7c-5be4-4f8c-a584-14f53dbfef70","Type":"ContainerDied","Data":"2f5b6861d4e61c36d93ec5647d308b10e0659da8d7b5b2d139b259cef362f208"} Oct 29 23:08:46.829489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.829048 5736 scope.go:122] "RemoveContainer" containerID="6682af431b576bf67a325b810713479738eaf4a1cb50525db42496a12011b2a8" Oct 29 23:08:46.830710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.830691 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" Oct 29 23:08:46.830844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.830719 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv" event={"ID":"88ecad2a-3820-48b3-ae06-ec40179ac8ca","Type":"ContainerDied","Data":"caee6a9ee079cdd55fcdb278d331dcc5f31d423073939305b77af448fba5b896"} Oct 29 23:08:46.830844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.830778 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caee6a9ee079cdd55fcdb278d331dcc5f31d423073939305b77af448fba5b896" Oct 29 23:08:46.909930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.909901 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd"] Oct 29 23:08:46.914339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:46.914316 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-hn6vd"] Oct 29 23:08:47.005324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:47.005294 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" path="/var/lib/kubelet/pods/44af3d7c-5be4-4f8c-a584-14f53dbfef70/volumes" Oct 29 23:08:47.005787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:47.005772 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07d200c-59c7-463b-af25-e6f4622f2294" path="/var/lib/kubelet/pods/e07d200c-59c7-463b-af25-e6f4622f2294/volumes" Oct 29 23:08:50.653801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:50.653766 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph"] Oct 29 23:08:53.316643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:53.307297 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262"] Oct 29 23:08:53.323604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:53.321209 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" containerID="cri-o://6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab" gracePeriod=30 Oct 29 23:08:55.654228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:55.654181 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" podUID="1946032b-9b2d-4b08-abaf-620d8da43823" containerName="payload-processing" containerID="cri-o://199a8c0e5163ee851121a4b134b66f6ddfc8d2c65724c8d819cbec9ab73aa8e6" gracePeriod=25 Oct 29 23:08:55.918660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:55.915783 5736 generic.go:356] "Generic (PLEG): container finished" podID="1946032b-9b2d-4b08-abaf-620d8da43823" containerID="199a8c0e5163ee851121a4b134b66f6ddfc8d2c65724c8d819cbec9ab73aa8e6" exitCode=0 Oct 29 23:08:55.918660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:55.915897 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" event={"ID":"1946032b-9b2d-4b08-abaf-620d8da43823","Type":"ContainerDied","Data":"199a8c0e5163ee851121a4b134b66f6ddfc8d2c65724c8d819cbec9ab73aa8e6"} Oct 29 23:08:56.045605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.045001 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:08:56.114089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.114059 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1946032b-9b2d-4b08-abaf-620d8da43823-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1946032b-9b2d-4b08-abaf-620d8da43823-plugins-config-volume\") pod \"1946032b-9b2d-4b08-abaf-620d8da43823\" (UID: \"1946032b-9b2d-4b08-abaf-620d8da43823\") " Oct 29 23:08:56.114479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.114461 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1946032b-9b2d-4b08-abaf-620d8da43823-kube-api-access-xnhw9\" (UniqueName: \"kubernetes.io/projected/1946032b-9b2d-4b08-abaf-620d8da43823-kube-api-access-xnhw9\") pod \"1946032b-9b2d-4b08-abaf-620d8da43823\" (UID: \"1946032b-9b2d-4b08-abaf-620d8da43823\") " Oct 29 23:08:56.117672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.117650 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1946032b-9b2d-4b08-abaf-620d8da43823-plugins-config-volume" pod "1946032b-9b2d-4b08-abaf-620d8da43823" (UID: "1946032b-9b2d-4b08-abaf-620d8da43823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:56.122567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.122548 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1946032b-9b2d-4b08-abaf-620d8da43823-kube-api-access-xnhw9" pod "1946032b-9b2d-4b08-abaf-620d8da43823" (UID: "1946032b-9b2d-4b08-abaf-620d8da43823"). InnerVolumeSpecName "kube-api-access-xnhw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:56.224602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.217215 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnhw9\" (UniqueName: \"kubernetes.io/projected/1946032b-9b2d-4b08-abaf-620d8da43823-kube-api-access-xnhw9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:56.224602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.217242 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1946032b-9b2d-4b08-abaf-620d8da43823-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:56.857949 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:08:56.857905 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:08:56.925643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.925609 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" Oct 29 23:08:56.925815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.925617 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph" event={"ID":"1946032b-9b2d-4b08-abaf-620d8da43823","Type":"ContainerDied","Data":"585daadfba1257e51c7065e3764a84ba9c7f8941e6d5dc74e28b80d12de40d8d"} Oct 29 23:08:56.925815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.925749 5736 scope.go:122] "RemoveContainer" containerID="199a8c0e5163ee851121a4b134b66f6ddfc8d2c65724c8d819cbec9ab73aa8e6" Oct 29 23:08:56.978769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.978734 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph"] Oct 29 23:08:56.991298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:56.991261 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd000a-6bd97759db-d89ph"] Oct 29 23:08:57.005367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:57.005329 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1946032b-9b2d-4b08-abaf-620d8da43823" path="/var/lib/kubelet/pods/1946032b-9b2d-4b08-abaf-620d8da43823/volumes" Oct 29 23:08:57.122195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:57.122072 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9"] Oct 29 23:08:57.125810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:57.125771 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57b28e-c67b9"] Oct 29 23:08:58.281485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.281442 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7"] Oct 29 23:08:58.282812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282785 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1946032b-9b2d-4b08-abaf-620d8da43823" containerName="payload-processing" Oct 29 23:08:58.282812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282811 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1946032b-9b2d-4b08-abaf-620d8da43823" containerName="payload-processing" Oct 29 23:08:58.282999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282841 5736 cpu_manager.go:414] "containerMap: removing container" podUID="adff96c3-569c-409a-8b5f-996e97e797d3" containerName="revoke-keys" Oct 29 23:08:58.282999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282849 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adff96c3-569c-409a-8b5f-996e97e797d3" containerName="revoke-keys" Oct 29 23:08:58.282999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282873 5736 cpu_manager.go:414] "containerMap: removing container" podUID="88ecad2a-3820-48b3-ae06-ec40179ac8ca" containerName="revoke-keys" Oct 29 23:08:58.282999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282880 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ecad2a-3820-48b3-ae06-ec40179ac8ca" containerName="revoke-keys" Oct 29 23:08:58.282999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282931 5736 cpu_manager.go:414] "containerMap: removing container" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" Oct 29 23:08:58.282999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282938 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" Oct 29 23:08:58.282999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282956 5736 cpu_manager.go:414] "containerMap: removing container" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" Oct 29 23:08:58.282999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.282962 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" Oct 29 23:08:58.283389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.283218 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cfaf69-db9f-4bee-ac82-b012b725172d" containerName="istio-proxy" Oct 29 23:08:58.283389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.283245 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1946032b-9b2d-4b08-abaf-620d8da43823" containerName="payload-processing" Oct 29 23:08:58.283389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.283265 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="adff96c3-569c-409a-8b5f-996e97e797d3" containerName="revoke-keys" Oct 29 23:08:58.283389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.283282 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ecad2a-3820-48b3-ae06-ec40179ac8ca" containerName="revoke-keys" Oct 29 23:08:58.283389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.283302 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="44af3d7c-5be4-4f8c-a584-14f53dbfef70" containerName="istio-proxy" Oct 29 23:08:58.289592 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.289553 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.302691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.302665 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7"] Oct 29 23:08:58.315622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.311820 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:58.315622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.311867 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:08:58.362779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.362749 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f016f2fd-4f6x7\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.362917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.362870 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vxf\" (UniqueName: \"kubernetes.io/projected/12c39fbe-50a0-44c5-abd7-f336ad491c85-kube-api-access-t4vxf\") pod \"maas-api-revoke-sub-f016f2fd-4f6x7\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.363271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.363251 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f016f2fd-4f6x7\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.464625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.464563 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f016f2fd-4f6x7\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.464749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.464698 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vxf\" (UniqueName: \"kubernetes.io/projected/12c39fbe-50a0-44c5-abd7-f336ad491c85-kube-api-access-t4vxf\") pod \"maas-api-revoke-sub-f016f2fd-4f6x7\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.464873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.464859 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f016f2fd-4f6x7\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.465708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.465679 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f016f2fd-4f6x7\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.466967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.466943 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f016f2fd-4f6x7\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.496475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.496440 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vxf\" (UniqueName: \"kubernetes.io/projected/12c39fbe-50a0-44c5-abd7-f336ad491c85-kube-api-access-t4vxf\") pod \"maas-api-revoke-sub-f016f2fd-4f6x7\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.586725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.586695 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:08:58.604762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.604735 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:08:58.668748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.667653 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-istio-token\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-istio-token\") pod \"9569f274-5c9b-4a99-81f2-7321183979b2\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " Oct 29 23:08:58.668748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.667779 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9569f274-5c9b-4a99-81f2-7321183979b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9569f274-5c9b-4a99-81f2-7321183979b2-istiod-ca-cert\") pod \"9569f274-5c9b-4a99-81f2-7321183979b2\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " Oct 29 23:08:58.668748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.667843 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9569f274-5c9b-4a99-81f2-7321183979b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9569f274-5c9b-4a99-81f2-7321183979b2-istio-podinfo\") pod \"9569f274-5c9b-4a99-81f2-7321183979b2\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " Oct 29 23:08:58.668748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.667890 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-kube-api-access-k59dz\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-kube-api-access-k59dz\") pod \"9569f274-5c9b-4a99-81f2-7321183979b2\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " Oct 29 23:08:58.668748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.667928 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-envoy\") pod \"9569f274-5c9b-4a99-81f2-7321183979b2\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " Oct 29 23:08:58.668748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.668007 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-data\") pod \"9569f274-5c9b-4a99-81f2-7321183979b2\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " Oct 29 23:08:58.668748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.668059 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-credential-socket\") pod \"9569f274-5c9b-4a99-81f2-7321183979b2\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " Oct 29 23:08:58.668748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.668100 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-socket\") pod \"9569f274-5c9b-4a99-81f2-7321183979b2\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " Oct 29 23:08:58.668748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.668203 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-certs\") pod \"9569f274-5c9b-4a99-81f2-7321183979b2\" (UID: \"9569f274-5c9b-4a99-81f2-7321183979b2\") " Oct 29 23:08:58.670720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.670086 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-data" pod "9569f274-5c9b-4a99-81f2-7321183979b2" (UID: "9569f274-5c9b-4a99-81f2-7321183979b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:58.670720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.670409 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9569f274-5c9b-4a99-81f2-7321183979b2-istiod-ca-cert" pod "9569f274-5c9b-4a99-81f2-7321183979b2" (UID: "9569f274-5c9b-4a99-81f2-7321183979b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:58.671343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.671178 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-credential-socket" pod "9569f274-5c9b-4a99-81f2-7321183979b2" (UID: "9569f274-5c9b-4a99-81f2-7321183979b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:58.671674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.671481 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-socket" pod "9569f274-5c9b-4a99-81f2-7321183979b2" (UID: "9569f274-5c9b-4a99-81f2-7321183979b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:58.672513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.671804 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-certs" pod "9569f274-5c9b-4a99-81f2-7321183979b2" (UID: "9569f274-5c9b-4a99-81f2-7321183979b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:58.672513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.672176 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:58.672513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.672200 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9569f274-5c9b-4a99-81f2-7321183979b2-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:58.672513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.672218 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:58.672513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.672236 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:58.672513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.672254 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:58.677273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.677241 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-envoy" pod "9569f274-5c9b-4a99-81f2-7321183979b2" (UID: "9569f274-5c9b-4a99-81f2-7321183979b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:58.678398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.678369 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9569f274-5c9b-4a99-81f2-7321183979b2-istio-podinfo" pod "9569f274-5c9b-4a99-81f2-7321183979b2" (UID: "9569f274-5c9b-4a99-81f2-7321183979b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:58.689031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.688997 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-kube-api-access-k59dz" pod "9569f274-5c9b-4a99-81f2-7321183979b2" (UID: "9569f274-5c9b-4a99-81f2-7321183979b2"). InnerVolumeSpecName "kube-api-access-k59dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:58.690921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.690885 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-istio-token" pod "9569f274-5c9b-4a99-81f2-7321183979b2" (UID: "9569f274-5c9b-4a99-81f2-7321183979b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:58.774386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.774352 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:58.774386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.774387 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9569f274-5c9b-4a99-81f2-7321183979b2-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:58.774624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.774401 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k59dz\" (UniqueName: \"kubernetes.io/projected/9569f274-5c9b-4a99-81f2-7321183979b2-kube-api-access-k59dz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:58.774624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.774416 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9569f274-5c9b-4a99-81f2-7321183979b2-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:08:58.908883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.908804 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7"] Oct 29 23:08:58.914439 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:08:58.914410 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c39fbe_50a0_44c5_abd7_f336ad491c85.slice/crio-4067bf5bab0564fcef58138a8b33a42052e046b67aa2ab729d16080448ec24e5 WatchSource:0}: Error finding container 4067bf5bab0564fcef58138a8b33a42052e046b67aa2ab729d16080448ec24e5: Status 404 returned error can't find the container with id 4067bf5bab0564fcef58138a8b33a42052e046b67aa2ab729d16080448ec24e5 Oct 29 23:08:58.947823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.947794 5736 generic.go:356] "Generic (PLEG): container finished" podID="9569f274-5c9b-4a99-81f2-7321183979b2" containerID="6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab" exitCode=0 Oct 29 23:08:58.947943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.947864 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" Oct 29 23:08:58.947943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.947878 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" event={"ID":"9569f274-5c9b-4a99-81f2-7321183979b2","Type":"ContainerDied","Data":"6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab"} Oct 29 23:08:58.947943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.947933 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262" event={"ID":"9569f274-5c9b-4a99-81f2-7321183979b2","Type":"ContainerDied","Data":"7db2d3caa176903b479496fb7ecf0f94675c509d550b201f2f4f46fed9bcf0fc"} Oct 29 23:08:58.948119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.947961 5736 scope.go:122] "RemoveContainer" containerID="6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab" Oct 29 23:08:58.949850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.949827 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" event={"ID":"12c39fbe-50a0-44c5-abd7-f336ad491c85","Type":"ContainerStarted","Data":"4067bf5bab0564fcef58138a8b33a42052e046b67aa2ab729d16080448ec24e5"} Oct 29 23:08:58.960328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.960301 5736 scope.go:122] "RemoveContainer" containerID="045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb" Oct 29 23:08:58.970991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.970972 5736 scope.go:122] "RemoveContainer" containerID="6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab" Oct 29 23:08:58.971300 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:08:58.971272 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab\": container with ID starting with 6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab not found: ID does not exist" containerID="6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab" Oct 29 23:08:58.971358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.971312 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab"} err="failed to get container status \"6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab\": rpc error: code = NotFound desc = could not find container \"6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab\": container with ID starting with 6fc3cdbf1d5cc5adeca139107503bcc2d6b2676a8501cf45a4cd7a6a0d8373ab not found: ID does not exist" Oct 29 23:08:58.971358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.971337 5736 scope.go:122] "RemoveContainer" containerID="045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb" Oct 29 23:08:58.971662 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:08:58.971639 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb\": container with ID starting with 045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb not found: ID does not exist" containerID="045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb" Oct 29 23:08:58.971763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:58.971664 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb"} err="failed to get container status \"045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb\": rpc error: code = NotFound desc = could not find container \"045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb\": container with ID starting with 045b93bd081347bb6365fb17f5a31e7cabeaf59a4fa92b9d939184aeb8c623bb not found: ID does not exist" Oct 29 23:08:59.005694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:59.005665 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b62de84-4b61-44e4-801f-48f3c7b7fbb4" path="/var/lib/kubelet/pods/3b62de84-4b61-44e4-801f-48f3c7b7fbb4/volumes" Oct 29 23:08:59.006084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:59.006063 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262"] Oct 29 23:08:59.012625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:59.012600 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-957eef-openshift-default-6f474857fc-tz262"] Oct 29 23:08:59.961897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:59.961861 5736 generic.go:356] "Generic (PLEG): container finished" podID="12c39fbe-50a0-44c5-abd7-f336ad491c85" containerID="731728df20f6118d7eb5f53f3d11e03ecac8a16d05040e09753ddc29c037b99c" exitCode=0 Oct 29 23:08:59.962330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:08:59.961944 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" event={"ID":"12c39fbe-50a0-44c5-abd7-f336ad491c85","Type":"ContainerDied","Data":"731728df20f6118d7eb5f53f3d11e03ecac8a16d05040e09753ddc29c037b99c"} Oct 29 23:09:01.007416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.007378 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" path="/var/lib/kubelet/pods/9569f274-5c9b-4a99-81f2-7321183979b2/volumes" Oct 29 23:09:01.149288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.149258 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:09:01.211190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.211159 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-xks-ca\") pod \"12c39fbe-50a0-44c5-abd7-f336ad491c85\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " Oct 29 23:09:01.211385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.211250 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c39fbe-50a0-44c5-abd7-f336ad491c85-kube-api-access-t4vxf\" (UniqueName: \"kubernetes.io/projected/12c39fbe-50a0-44c5-abd7-f336ad491c85-kube-api-access-t4vxf\") pod \"12c39fbe-50a0-44c5-abd7-f336ad491c85\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " Oct 29 23:09:01.211497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.211478 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-openshift-ca\") pod \"12c39fbe-50a0-44c5-abd7-f336ad491c85\" (UID: \"12c39fbe-50a0-44c5-abd7-f336ad491c85\") " Oct 29 23:09:01.212028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.211938 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-openshift-ca" pod "12c39fbe-50a0-44c5-abd7-f336ad491c85" (UID: "12c39fbe-50a0-44c5-abd7-f336ad491c85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:01.212103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.212083 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:01.213910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.213884 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-xks-ca" pod "12c39fbe-50a0-44c5-abd7-f336ad491c85" (UID: "12c39fbe-50a0-44c5-abd7-f336ad491c85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:09:01.214008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.213992 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c39fbe-50a0-44c5-abd7-f336ad491c85-kube-api-access-t4vxf" pod "12c39fbe-50a0-44c5-abd7-f336ad491c85" (UID: "12c39fbe-50a0-44c5-abd7-f336ad491c85"). InnerVolumeSpecName "kube-api-access-t4vxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:01.313370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.313334 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12c39fbe-50a0-44c5-abd7-f336ad491c85-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:01.313370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.313366 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4vxf\" (UniqueName: \"kubernetes.io/projected/12c39fbe-50a0-44c5-abd7-f336ad491c85-kube-api-access-t4vxf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:01.979278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.979251 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" Oct 29 23:09:01.979518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.979288 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7" event={"ID":"12c39fbe-50a0-44c5-abd7-f336ad491c85","Type":"ContainerDied","Data":"4067bf5bab0564fcef58138a8b33a42052e046b67aa2ab729d16080448ec24e5"} Oct 29 23:09:01.979518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:01.979331 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4067bf5bab0564fcef58138a8b33a42052e046b67aa2ab729d16080448ec24e5" Oct 29 23:09:14.193379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:14.193293 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d"] Oct 29 23:09:14.212262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:14.212215 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd000a-snj8d"] Oct 29 23:09:15.012312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:15.012272 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adff96c3-569c-409a-8b5f-996e97e797d3" path="/var/lib/kubelet/pods/adff96c3-569c-409a-8b5f-996e97e797d3/volumes" Oct 29 23:09:15.012832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:15.012803 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv"] Oct 29 23:09:15.017159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:15.017118 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c23236-ntkzv"] Oct 29 23:09:17.004215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:17.004182 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ecad2a-3820-48b3-ae06-ec40179ac8ca" path="/var/lib/kubelet/pods/88ecad2a-3820-48b3-ae06-ec40179ac8ca/volumes" Oct 29 23:09:17.761174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:17.761139 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh"] Oct 29 23:09:17.761404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:17.761368 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" containerID="cri-o://e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5" gracePeriod=30 Oct 29 23:09:22.933216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:22.933173 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 23:09:22.933794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:22.933245 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 23:09:23.072284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.072228 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 23:09:23.161255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.161221 5736 generic.go:356] "Generic (PLEG): container finished" podID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerID="e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5" exitCode=0 Oct 29 23:09:23.161417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.161285 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" Oct 29 23:09:23.161417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.161311 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" event={"ID":"3fdb0288-46c8-4c6e-a350-beccdd8b6a90","Type":"ContainerDied","Data":"e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5"} Oct 29 23:09:23.161417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.161383 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh" event={"ID":"3fdb0288-46c8-4c6e-a350-beccdd8b6a90","Type":"ContainerDied","Data":"93ae5beba1e8e88ba7e32089d8208c77d84563d0d62a6a227175d2c85e5b0e81"} Oct 29 23:09:23.161527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.161420 5736 scope.go:122] "RemoveContainer" containerID="e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5" Oct 29 23:09:23.172474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.172440 5736 scope.go:122] "RemoveContainer" containerID="e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5" Oct 29 23:09:23.172818 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:09:23.172789 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5\": container with ID starting with e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5 not found: ID does not exist" containerID="e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5" Oct 29 23:09:23.172952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.172829 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5"} err="failed to get container status \"e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5\": rpc error: code = NotFound desc = could not find container \"e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5\": container with ID starting with e72fadabf02c15ad619064e4fe4388d8d6ee713db85ac1ac225dc702d579d1d5 not found: ID does not exist" Oct 29 23:09:23.197487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.197460 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-data\") pod \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " Oct 29 23:09:23.197670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.197549 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-socket\") pod \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " Oct 29 23:09:23.197670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.197611 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-envoy\") pod \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " Oct 29 23:09:23.197670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.197642 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-credential-socket\") pod \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " Oct 29 23:09:23.197850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.197741 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istiod-ca-cert\") pod \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " Oct 29 23:09:23.197850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.197757 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-data" pod "3fdb0288-46c8-4c6e-a350-beccdd8b6a90" (UID: "3fdb0288-46c8-4c6e-a350-beccdd8b6a90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:23.197850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.197777 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-certs\") pod \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " Oct 29 23:09:23.198000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.197873 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-socket" pod "3fdb0288-46c8-4c6e-a350-beccdd8b6a90" (UID: "3fdb0288-46c8-4c6e-a350-beccdd8b6a90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:23.198000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.197970 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-podinfo\") pod \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " Oct 29 23:09:23.198108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.198019 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-kube-api-access-g9hzq\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-kube-api-access-g9hzq\") pod \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " Oct 29 23:09:23.198108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.198045 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-credential-socket" pod "3fdb0288-46c8-4c6e-a350-beccdd8b6a90" (UID: "3fdb0288-46c8-4c6e-a350-beccdd8b6a90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:23.198108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.198057 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-token\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-token\") pod \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\" (UID: \"3fdb0288-46c8-4c6e-a350-beccdd8b6a90\") " Oct 29 23:09:23.198272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.198179 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istiod-ca-cert" pod "3fdb0288-46c8-4c6e-a350-beccdd8b6a90" (UID: "3fdb0288-46c8-4c6e-a350-beccdd8b6a90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:23.198272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.198187 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-certs" pod "3fdb0288-46c8-4c6e-a350-beccdd8b6a90" (UID: "3fdb0288-46c8-4c6e-a350-beccdd8b6a90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:23.198983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.198957 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:23.199141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.198988 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:23.199141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.199004 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:23.199141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.199019 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:23.199141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.199032 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:23.200198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.200171 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-envoy" pod "3fdb0288-46c8-4c6e-a350-beccdd8b6a90" (UID: "3fdb0288-46c8-4c6e-a350-beccdd8b6a90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:09:23.200550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.200530 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-token" pod "3fdb0288-46c8-4c6e-a350-beccdd8b6a90" (UID: "3fdb0288-46c8-4c6e-a350-beccdd8b6a90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:23.200719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.200702 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-kube-api-access-g9hzq" pod "3fdb0288-46c8-4c6e-a350-beccdd8b6a90" (UID: "3fdb0288-46c8-4c6e-a350-beccdd8b6a90"). InnerVolumeSpecName "kube-api-access-g9hzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:23.200979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.200951 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-podinfo" pod "3fdb0288-46c8-4c6e-a350-beccdd8b6a90" (UID: "3fdb0288-46c8-4c6e-a350-beccdd8b6a90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:09:23.300806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.300778 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:23.300806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.300804 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9hzq\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-kube-api-access-g9hzq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:23.301002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.300816 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:23.301002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.300824 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdb0288-46c8-4c6e-a350-beccdd8b6a90-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:09:23.507782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.507754 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh"] Oct 29 23:09:23.519700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:23.519671 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-pwkhh"] Oct 29 23:09:24.551175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:24.551139 5736 scope.go:122] "RemoveContainer" containerID="7829bd7de50ae15f74742fd722be0f7566c8e3fa45f1298d0b645f2e15b11e17" Oct 29 23:09:24.639565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:24.639359 5736 scope.go:122] "RemoveContainer" containerID="aff0b072f6a5a80db99b9af6779e33bb4c8e4a8911d2c23aa59c9052516b2dd8" Oct 29 23:09:25.005715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:09:25.005687 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" path="/var/lib/kubelet/pods/3fdb0288-46c8-4c6e-a350-beccdd8b6a90/volumes" Oct 29 23:09:26.824903 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:09:26.824865 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:09:56.824859 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:09:56.824811 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:10:02.930590 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.930548 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj"] Oct 29 23:10:02.931441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.931416 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" Oct 29 23:10:02.931635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.931618 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" Oct 29 23:10:02.931759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.931747 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" Oct 29 23:10:02.931913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.931897 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" Oct 29 23:10:02.932064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.932053 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" Oct 29 23:10:02.932160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.932148 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" Oct 29 23:10:02.932249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.932239 5736 cpu_manager.go:414] "containerMap: removing container" podUID="12c39fbe-50a0-44c5-abd7-f336ad491c85" containerName="revoke-keys" Oct 29 23:10:02.932324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.932313 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c39fbe-50a0-44c5-abd7-f336ad491c85" containerName="revoke-keys" Oct 29 23:10:02.932607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.932574 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdb0288-46c8-4c6e-a350-beccdd8b6a90" containerName="istio-proxy" Oct 29 23:10:02.932724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.932712 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" Oct 29 23:10:02.932822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.932811 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c39fbe-50a0-44c5-abd7-f336ad491c85" containerName="revoke-keys" Oct 29 23:10:02.936915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.936893 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:02.986639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:02.986595 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj"] Oct 29 23:10:03.091185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.091148 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.091369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.091224 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.091428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.091391 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.091480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.091450 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-kube-api-access-p4j78\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.091661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.091635 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.091749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.091687 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.091795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.091770 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d85aebb-8ada-4414-8663-4af63145e5ce-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.091844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.091811 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.091957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.091939 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.192973 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193020 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193050 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d85aebb-8ada-4414-8663-4af63145e5ce-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193068 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193134 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193247 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193304 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193383 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193427 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-kube-api-access-p4j78\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193434 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193822 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193879 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d85aebb-8ada-4414-8663-4af63145e5ce-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.193986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193917 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.194045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.193994 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.195532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.195501 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.195796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.195775 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.217471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.217439 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.217636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.217597 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-kube-api-access-p4j78\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.287556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.287522 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:03.646642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.646616 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj"] Oct 29 23:10:03.648907 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:03.648873 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d85aebb_8ada_4414_8663_4af63145e5ce.slice/crio-9e4c646f1eabfc16ea07e94d7af76a4a5ad2caf8f3199f1b51e5dab89b48cc7b WatchSource:0}: Error finding container 9e4c646f1eabfc16ea07e94d7af76a4a5ad2caf8f3199f1b51e5dab89b48cc7b: Status 404 returned error can't find the container with id 9e4c646f1eabfc16ea07e94d7af76a4a5ad2caf8f3199f1b51e5dab89b48cc7b Oct 29 23:10:03.651516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.651480 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:03.651644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.651552 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:03.651644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:03.651597 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:04.508980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:04.506008 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" event={"ID":"0d85aebb-8ada-4414-8663-4af63145e5ce","Type":"ContainerStarted","Data":"7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df"} Oct 29 23:10:04.508980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:04.506074 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" event={"ID":"0d85aebb-8ada-4414-8663-4af63145e5ce","Type":"ContainerStarted","Data":"9e4c646f1eabfc16ea07e94d7af76a4a5ad2caf8f3199f1b51e5dab89b48cc7b"} Oct 29 23:10:04.569601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:04.568680 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podStartSLOduration=2.5686430810000003 podStartE2EDuration="2.568643081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:04.566205119 +0000 UTC m=+1601.682124725" watchObservedRunningTime="2026-10-29 23:10:04.568643081 +0000 UTC m=+1601.684562690" Oct 29 23:10:05.290484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.290442 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:05.290938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.290906 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:10:05.291189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.291144 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 23:10:05.845871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.845814 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.903320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.903272 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.904193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.904130 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.947840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.947786 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.948502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.948469 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 23:10:05.960561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.960520 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 23:10:05.984416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.982893 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.984416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:05.983759 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.007722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.003158 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.063790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.063744 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" podUID="db18ea92-9d40-4840-9c15-353d841dea31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/health\": context deadline exceeded" Oct 29 23:10:06.249727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.249672 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.297045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.297007 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:10:06.297376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.297336 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 23:10:06.351938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.351897 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.352524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.352491 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 23:10:06.353818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.353108 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 29 23:10:06.399719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.399673 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.485483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.485438 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.546557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.546459 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 29 23:10:06.665154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.566719 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.762672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.762635 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6"] Oct 29 23:10:06.763180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.763143 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" containerID="cri-o://6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f" gracePeriod=30 Oct 29 23:10:06.782998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.782960 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" podUID="db18ea92-9d40-4840-9c15-353d841dea31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.840404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.816266 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 29 23:10:06.886679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.886634 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v"] Oct 29 23:10:06.887451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.887414 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" containerID="cri-o://2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6" gracePeriod=30 Oct 29 23:10:06.888974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.888933 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.890222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.889303 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:10:06.907957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.907921 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.938014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.937969 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:06.939488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.939465 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 23:10:06.996144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.995042 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 23:10:06.996846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:06.996814 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 23:10:07.017150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.017106 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 23:10:07.059604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.017429 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:10:07.060234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.060207 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:10:07.060434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.060419 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:10:07.081604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.024742 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 23:10:07.093231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.073524 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 23:10:07.094754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.024684 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 29 23:10:07.107266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.107219 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:10:07.107534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.107519 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 23:10:07.107693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.107680 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:10:07.107827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.107817 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:10:07.340614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.336511 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:10:07.340910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.340873 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 23:10:07.600612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.599695 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 23:10:07.604604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.601990 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 23:10:07.641600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.638551 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 23:10:07.641600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.638683 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:07.654414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.654374 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:07.655808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.655780 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 29 23:10:07.656722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.656700 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 23:10:07.656898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.656884 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:10:07.657031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.657020 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:10:07.672601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.654524 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 29 23:10:07.755811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.654554 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 23:10:07.779538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.676329 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:10:07.779538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.756789 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:10:07.779538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.756934 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:10:07.779538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.757067 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:10:07.779538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.757106 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 23:10:07.779538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.757241 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 23:10:07.779538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.721800 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 29 23:10:07.779538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.722903 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:07.779538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.734656 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 29 23:10:07.780246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.779673 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:10:07.838097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.838064 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:10:07.914517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.914420 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:07.915426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.915403 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 23:10:07.997535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:07.988651 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:08.029636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.026198 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:10:08.094734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.094683 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:08.103683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.103636 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:08.143898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.143863 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:10:08.162611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.161608 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 23:10:08.224831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.224803 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 23:10:08.273606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.271141 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 23:10:08.290714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.290687 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:10:08.294500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.294466 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 23:10:08.294774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.292732 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:10:08.296891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:08.296870 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 23:10:09.288554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:09.288521 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:10:09.288969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:09.288592 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 23:10:10.053600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.052689 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt"] Oct 29 23:10:10.057097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.057071 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9569f274-5c9b-4a99-81f2-7321183979b2" containerName="istio-proxy" Oct 29 23:10:10.061053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.060813 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.063051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.063029 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt"] Oct 29 23:10:10.075859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.075839 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:10:10.087946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.087923 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:10:10.108050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.108024 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.108192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.108165 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.108491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.108466 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgjpr\" (UniqueName: \"kubernetes.io/projected/c728b3ea-1482-4a9e-8582-f8ea6a448f14-kube-api-access-cgjpr\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.209443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.209402 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgjpr\" (UniqueName: \"kubernetes.io/projected/c728b3ea-1482-4a9e-8582-f8ea6a448f14-kube-api-access-cgjpr\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.209673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.209473 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.209673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.209535 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.210276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.210251 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.211906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.211879 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.262992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.262956 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgjpr\" (UniqueName: \"kubernetes.io/projected/c728b3ea-1482-4a9e-8582-f8ea6a448f14-kube-api-access-cgjpr\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.288526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.288495 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:10:10.288705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.288553 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 23:10:10.378024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.377937 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:10.707933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.707838 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt"] Oct 29 23:10:10.819436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:10.819406 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" event={"ID":"c728b3ea-1482-4a9e-8582-f8ea6a448f14","Type":"ContainerStarted","Data":"fee50ad67f47a53926e0af000411ef5fe32d77bec0f7d87676987d8d3e293614"} Oct 29 23:10:11.288023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:11.287992 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:10:11.288193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:11.288045 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 23:10:11.829935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:11.829901 5736 generic.go:356] "Generic (PLEG): container finished" podID="c728b3ea-1482-4a9e-8582-f8ea6a448f14" containerID="775ca79b60f63559089a36141b242fe7057095843b938d66597c4c7212245f58" exitCode=0 Oct 29 23:10:11.830380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:11.829982 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" event={"ID":"c728b3ea-1482-4a9e-8582-f8ea6a448f14","Type":"ContainerDied","Data":"775ca79b60f63559089a36141b242fe7057095843b938d66597c4c7212245f58"} Oct 29 23:10:12.160162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.158510 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-ff84v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 29 23:10:12.160162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.159024 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 29 23:10:12.256715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.254610 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 23:10:12.351244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.351212 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 23:10:12.384883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.384852 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/596e957b-f674-4a0e-92eb-08913a45b069-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596e957b-f674-4a0e-92eb-08913a45b069-istio-podinfo\") pod \"596e957b-f674-4a0e-92eb-08913a45b069\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " Oct 29 23:10:12.385034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.384904 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-istio-token\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-istio-token\") pod \"596e957b-f674-4a0e-92eb-08913a45b069\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " Oct 29 23:10:12.385034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.384985 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-data\") pod \"596e957b-f674-4a0e-92eb-08913a45b069\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " Oct 29 23:10:12.385176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385105 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-envoy\") pod \"596e957b-f674-4a0e-92eb-08913a45b069\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " Oct 29 23:10:12.385271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385246 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-credential-socket\") pod \"596e957b-f674-4a0e-92eb-08913a45b069\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " Oct 29 23:10:12.385271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385261 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-data" pod "596e957b-f674-4a0e-92eb-08913a45b069" (UID: "596e957b-f674-4a0e-92eb-08913a45b069"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.385455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385335 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-certs\") pod \"596e957b-f674-4a0e-92eb-08913a45b069\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " Oct 29 23:10:12.385455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385395 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596e957b-f674-4a0e-92eb-08913a45b069-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596e957b-f674-4a0e-92eb-08913a45b069-istiod-ca-cert\") pod \"596e957b-f674-4a0e-92eb-08913a45b069\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " Oct 29 23:10:12.385455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385433 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-kube-api-access-6hc8q\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-kube-api-access-6hc8q\") pod \"596e957b-f674-4a0e-92eb-08913a45b069\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " Oct 29 23:10:12.385682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385542 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-credential-socket" pod "596e957b-f674-4a0e-92eb-08913a45b069" (UID: "596e957b-f674-4a0e-92eb-08913a45b069"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.385682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385566 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-socket\") pod \"596e957b-f674-4a0e-92eb-08913a45b069\" (UID: \"596e957b-f674-4a0e-92eb-08913a45b069\") " Oct 29 23:10:12.385800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385704 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-certs" pod "596e957b-f674-4a0e-92eb-08913a45b069" (UID: "596e957b-f674-4a0e-92eb-08913a45b069"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.385914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.385887 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596e957b-f674-4a0e-92eb-08913a45b069-istiod-ca-cert" pod "596e957b-f674-4a0e-92eb-08913a45b069" (UID: "596e957b-f674-4a0e-92eb-08913a45b069"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:12.386177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.386153 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-socket" pod "596e957b-f674-4a0e-92eb-08913a45b069" (UID: "596e957b-f674-4a0e-92eb-08913a45b069"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.386346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.386322 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.386428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.386349 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.386428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.386364 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.386428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.386378 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.386428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.386391 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596e957b-f674-4a0e-92eb-08913a45b069-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.387372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.387350 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/596e957b-f674-4a0e-92eb-08913a45b069-istio-podinfo" pod "596e957b-f674-4a0e-92eb-08913a45b069" (UID: "596e957b-f674-4a0e-92eb-08913a45b069"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:10:12.387474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.387402 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-istio-token" pod "596e957b-f674-4a0e-92eb-08913a45b069" (UID: "596e957b-f674-4a0e-92eb-08913a45b069"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:12.388127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.388108 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-kube-api-access-6hc8q" pod "596e957b-f674-4a0e-92eb-08913a45b069" (UID: "596e957b-f674-4a0e-92eb-08913a45b069"). InnerVolumeSpecName "kube-api-access-6hc8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:12.388673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.388650 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-envoy" pod "596e957b-f674-4a0e-92eb-08913a45b069" (UID: "596e957b-f674-4a0e-92eb-08913a45b069"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.487382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487344 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-socket\") pod \"2d23e612-818a-43b7-9ff1-c3d37820460f\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487398 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-credential-socket\") pod \"2d23e612-818a-43b7-9ff1-c3d37820460f\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487430 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-podinfo\") pod \"2d23e612-818a-43b7-9ff1-c3d37820460f\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487510 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d23e612-818a-43b7-9ff1-c3d37820460f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d23e612-818a-43b7-9ff1-c3d37820460f-istiod-ca-cert\") pod \"2d23e612-818a-43b7-9ff1-c3d37820460f\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487709 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-socket" pod "2d23e612-818a-43b7-9ff1-c3d37820460f" (UID: "2d23e612-818a-43b7-9ff1-c3d37820460f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487715 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-token\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-token\") pod \"2d23e612-818a-43b7-9ff1-c3d37820460f\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487776 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-credential-socket" pod "2d23e612-818a-43b7-9ff1-c3d37820460f" (UID: "2d23e612-818a-43b7-9ff1-c3d37820460f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487887 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-certs\") pod \"2d23e612-818a-43b7-9ff1-c3d37820460f\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487976 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-kube-api-access-sg2z5\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-kube-api-access-sg2z5\") pod \"2d23e612-818a-43b7-9ff1-c3d37820460f\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488032 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-data\") pod \"2d23e612-818a-43b7-9ff1-c3d37820460f\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " Oct 29 23:10:12.488132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488115 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-envoy\") pod \"2d23e612-818a-43b7-9ff1-c3d37820460f\" (UID: \"2d23e612-818a-43b7-9ff1-c3d37820460f\") " Oct 29 23:10:12.488768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.487973 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d23e612-818a-43b7-9ff1-c3d37820460f-istiod-ca-cert" pod "2d23e612-818a-43b7-9ff1-c3d37820460f" (UID: "2d23e612-818a-43b7-9ff1-c3d37820460f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:12.488768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488222 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-certs" pod "2d23e612-818a-43b7-9ff1-c3d37820460f" (UID: "2d23e612-818a-43b7-9ff1-c3d37820460f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.488768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488441 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-data" pod "2d23e612-818a-43b7-9ff1-c3d37820460f" (UID: "2d23e612-818a-43b7-9ff1-c3d37820460f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.488932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488874 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.488932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488898 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.488932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488913 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d23e612-818a-43b7-9ff1-c3d37820460f-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.488932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488927 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596e957b-f674-4a0e-92eb-08913a45b069-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.489153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488940 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.489153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488953 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596e957b-f674-4a0e-92eb-08913a45b069-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.489153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488965 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.489153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488977 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hc8q\" (UniqueName: \"kubernetes.io/projected/596e957b-f674-4a0e-92eb-08913a45b069-kube-api-access-6hc8q\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.489153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.488991 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.490217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.490195 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-token" pod "2d23e612-818a-43b7-9ff1-c3d37820460f" (UID: "2d23e612-818a-43b7-9ff1-c3d37820460f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:12.490384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.490365 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-podinfo" pod "2d23e612-818a-43b7-9ff1-c3d37820460f" (UID: "2d23e612-818a-43b7-9ff1-c3d37820460f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:10:12.490574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.490556 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-kube-api-access-sg2z5" pod "2d23e612-818a-43b7-9ff1-c3d37820460f" (UID: "2d23e612-818a-43b7-9ff1-c3d37820460f"). InnerVolumeSpecName "kube-api-access-sg2z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:12.490886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.490866 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-envoy" pod "2d23e612-818a-43b7-9ff1-c3d37820460f" (UID: "2d23e612-818a-43b7-9ff1-c3d37820460f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:12.590140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.590065 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.590140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.590102 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg2z5\" (UniqueName: \"kubernetes.io/projected/2d23e612-818a-43b7-9ff1-c3d37820460f-kube-api-access-sg2z5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.590140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.590118 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.590140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.590132 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d23e612-818a-43b7-9ff1-c3d37820460f-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:12.840328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.840240 5736 generic.go:356] "Generic (PLEG): container finished" podID="596e957b-f674-4a0e-92eb-08913a45b069" containerID="6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f" exitCode=0 Oct 29 23:10:12.840328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.840293 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" Oct 29 23:10:12.840888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.840328 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" event={"ID":"596e957b-f674-4a0e-92eb-08913a45b069","Type":"ContainerDied","Data":"6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f"} Oct 29 23:10:12.840888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.840396 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" event={"ID":"596e957b-f674-4a0e-92eb-08913a45b069","Type":"ContainerDied","Data":"eb143e6c3789d3a0a932187b53156e701a90d829956409ac156fc40d77a684b4"} Oct 29 23:10:12.840888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.840430 5736 scope.go:122] "RemoveContainer" containerID="6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f" Oct 29 23:10:12.842226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.842196 5736 generic.go:356] "Generic (PLEG): container finished" podID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerID="2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6" exitCode=0 Oct 29 23:10:12.842362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.842242 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" event={"ID":"2d23e612-818a-43b7-9ff1-c3d37820460f","Type":"ContainerDied","Data":"2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6"} Oct 29 23:10:12.842362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.842320 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" event={"ID":"2d23e612-818a-43b7-9ff1-c3d37820460f","Type":"ContainerDied","Data":"aa1726ff71815ad85950c7062c59436c0dd35434ead63c248d6f28fb1443f64c"} Oct 29 23:10:12.842362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.842251 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v" Oct 29 23:10:12.853266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.853244 5736 scope.go:122] "RemoveContainer" containerID="6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f" Oct 29 23:10:12.853643 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:10:12.853615 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f\": container with ID starting with 6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f not found: ID does not exist" containerID="6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f" Oct 29 23:10:12.853741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.853649 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f"} err="failed to get container status \"6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f\": rpc error: code = NotFound desc = could not find container \"6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f\": container with ID starting with 6af0baff9663bc5d785c0e2b0c429d991d8380e469c9320f70347cdff4f5567f not found: ID does not exist" Oct 29 23:10:12.853741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.853671 5736 scope.go:122] "RemoveContainer" containerID="2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6" Oct 29 23:10:12.864406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.864388 5736 scope.go:122] "RemoveContainer" containerID="2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6" Oct 29 23:10:12.864880 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:10:12.864850 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6\": container with ID starting with 2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6 not found: ID does not exist" containerID="2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6" Oct 29 23:10:12.865000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.864884 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6"} err="failed to get container status \"2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6\": rpc error: code = NotFound desc = could not find container \"2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6\": container with ID starting with 2250bcd381d0c2032905c979341f85f41de08cdcab1c43ecdeee398bdc9753d6 not found: ID does not exist" Oct 29 23:10:12.896570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.896538 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6"] Oct 29 23:10:12.909404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.909368 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6"] Oct 29 23:10:12.943320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.943282 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v"] Oct 29 23:10:12.953057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:12.953024 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-ff84v"] Oct 29 23:10:13.007850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.007819 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" path="/var/lib/kubelet/pods/2d23e612-818a-43b7-9ff1-c3d37820460f/volumes" Oct 29 23:10:13.008338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.008320 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596e957b-f674-4a0e-92eb-08913a45b069" path="/var/lib/kubelet/pods/596e957b-f674-4a0e-92eb-08913a45b069/volumes" Oct 29 23:10:13.009670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.009650 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:13.096879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.096784 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c728b3ea-1482-4a9e-8582-f8ea6a448f14-kube-api-access-cgjpr\" (UniqueName: \"kubernetes.io/projected/c728b3ea-1482-4a9e-8582-f8ea6a448f14-kube-api-access-cgjpr\") pod \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " Oct 29 23:10:13.096879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.096844 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-xks-ca\") pod \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " Oct 29 23:10:13.097100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.096981 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-openshift-ca\") pod \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\" (UID: \"c728b3ea-1482-4a9e-8582-f8ea6a448f14\") " Oct 29 23:10:13.097390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.097361 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-openshift-ca" pod "c728b3ea-1482-4a9e-8582-f8ea6a448f14" (UID: "c728b3ea-1482-4a9e-8582-f8ea6a448f14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:13.097741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.097720 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.099020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.098990 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c728b3ea-1482-4a9e-8582-f8ea6a448f14-kube-api-access-cgjpr" pod "c728b3ea-1482-4a9e-8582-f8ea6a448f14" (UID: "c728b3ea-1482-4a9e-8582-f8ea6a448f14"). InnerVolumeSpecName "kube-api-access-cgjpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:13.099360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.099340 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-xks-ca" pod "c728b3ea-1482-4a9e-8582-f8ea6a448f14" (UID: "c728b3ea-1482-4a9e-8582-f8ea6a448f14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:13.135044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.135010 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-qmkl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:10:13.135215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.135063 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-qmkl6" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:13.199275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.199233 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgjpr\" (UniqueName: \"kubernetes.io/projected/c728b3ea-1482-4a9e-8582-f8ea6a448f14-kube-api-access-cgjpr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.199275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.199273 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c728b3ea-1482-4a9e-8582-f8ea6a448f14-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:13.328728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.328685 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:10:13.328904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.328749 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:13.853874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.853840 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" Oct 29 23:10:13.854402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.853871 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt" event={"ID":"c728b3ea-1482-4a9e-8582-f8ea6a448f14","Type":"ContainerDied","Data":"fee50ad67f47a53926e0af000411ef5fe32d77bec0f7d87676987d8d3e293614"} Oct 29 23:10:13.854402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:13.853964 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fee50ad67f47a53926e0af000411ef5fe32d77bec0f7d87676987d8d3e293614" Oct 29 23:10:14.055110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.055080 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:14.055628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.055571 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:14.056397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.056377 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:10:14.435982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.435947 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7"] Oct 29 23:10:14.436486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.436472 5736 cpu_manager.go:414] "containerMap: removing container" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" Oct 29 23:10:14.436540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.436487 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" Oct 29 23:10:14.436540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.436502 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c728b3ea-1482-4a9e-8582-f8ea6a448f14" containerName="revoke-keys" Oct 29 23:10:14.436540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.436508 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c728b3ea-1482-4a9e-8582-f8ea6a448f14" containerName="revoke-keys" Oct 29 23:10:14.436540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.436534 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" Oct 29 23:10:14.436540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.436539 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" Oct 29 23:10:14.436709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.436650 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c728b3ea-1482-4a9e-8582-f8ea6a448f14" containerName="revoke-keys" Oct 29 23:10:14.436709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.436666 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d23e612-818a-43b7-9ff1-c3d37820460f" containerName="istio-proxy" Oct 29 23:10:14.436709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.436676 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="596e957b-f674-4a0e-92eb-08913a45b069" containerName="istio-proxy" Oct 29 23:10:14.440071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.440046 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.456067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.456041 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:10:14.456185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.456142 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:10:14.460141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.460115 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7"] Oct 29 23:10:14.517675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.517644 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.517807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.517762 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.517850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.517817 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/5c3d8c66-cdba-430d-8644-2c7a0a893f94-kube-api-access-cqh8x\") pod \"maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.618526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.618497 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.618678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.618533 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/5c3d8c66-cdba-430d-8644-2c7a0a893f94-kube-api-access-cqh8x\") pod \"maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.618731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.618699 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.619164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.619145 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.620976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.620953 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.641371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.641345 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/5c3d8c66-cdba-430d-8644-2c7a0a893f94-kube-api-access-cqh8x\") pod \"maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:14.755898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:14.755872 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:15.060691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:15.060537 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7"] Oct 29 23:10:15.063865 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:15.063830 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3d8c66_cdba_430d_8644_2c7a0a893f94.slice/crio-2292e9247569d376e7e174dc06ce30b967e347314793f594b19ed80fb44ecf91 WatchSource:0}: Error finding container 2292e9247569d376e7e174dc06ce30b967e347314793f594b19ed80fb44ecf91: Status 404 returned error can't find the container with id 2292e9247569d376e7e174dc06ce30b967e347314793f594b19ed80fb44ecf91 Oct 29 23:10:15.884237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:15.884079 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" event={"ID":"5c3d8c66-cdba-430d-8644-2c7a0a893f94","Type":"ContainerStarted","Data":"df6db142fbba4f24054b1fe4820f4b2dd622635d6078ffd9fdca74c9a5c4a40f"} Oct 29 23:10:15.884237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:15.884191 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" event={"ID":"5c3d8c66-cdba-430d-8644-2c7a0a893f94","Type":"ContainerStarted","Data":"2292e9247569d376e7e174dc06ce30b967e347314793f594b19ed80fb44ecf91"} Oct 29 23:10:15.925594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:15.925538 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" podStartSLOduration=1.92552654 podStartE2EDuration="1.92552654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:15.924983438 +0000 UTC m=+1613.040903046" watchObservedRunningTime="2026-10-29 23:10:15.92552654 +0000 UTC m=+1613.041446146" Oct 29 23:10:16.891497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:16.891462 5736 generic.go:356] "Generic (PLEG): container finished" podID="5c3d8c66-cdba-430d-8644-2c7a0a893f94" containerID="df6db142fbba4f24054b1fe4820f4b2dd622635d6078ffd9fdca74c9a5c4a40f" exitCode=0 Oct 29 23:10:16.891913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:16.891563 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" event={"ID":"5c3d8c66-cdba-430d-8644-2c7a0a893f94","Type":"ContainerDied","Data":"df6db142fbba4f24054b1fe4820f4b2dd622635d6078ffd9fdca74c9a5c4a40f"} Oct 29 23:10:17.125184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.125151 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh"] Oct 29 23:10:17.129186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.129165 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.152245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.152187 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh"] Oct 29 23:10:17.259849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.259818 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2fsj\" (UniqueName: \"kubernetes.io/projected/51965f81-04ba-4a8b-9bea-ce56e06e7316-kube-api-access-f2fsj\") pod \"maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.260017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.259890 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.260017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.259961 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.361749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.361693 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.361957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.361776 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.362144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.362123 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2fsj\" (UniqueName: \"kubernetes.io/projected/51965f81-04ba-4a8b-9bea-ce56e06e7316-kube-api-access-f2fsj\") pod \"maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.362510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.362488 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.364272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.364247 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.385493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.385469 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2fsj\" (UniqueName: \"kubernetes.io/projected/51965f81-04ba-4a8b-9bea-ce56e06e7316-kube-api-access-f2fsj\") pod \"maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.442907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.442853 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:17.760140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.760106 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh"] Oct 29 23:10:17.763754 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:17.763725 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51965f81_04ba_4a8b_9bea_ce56e06e7316.slice/crio-071800635ee9d112a1f99878c54522621a94135dfe6322a9640f23e99e6ff4a1 WatchSource:0}: Error finding container 071800635ee9d112a1f99878c54522621a94135dfe6322a9640f23e99e6ff4a1: Status 404 returned error can't find the container with id 071800635ee9d112a1f99878c54522621a94135dfe6322a9640f23e99e6ff4a1 Oct 29 23:10:17.899390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:17.899353 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" event={"ID":"51965f81-04ba-4a8b-9bea-ce56e06e7316","Type":"ContainerStarted","Data":"071800635ee9d112a1f99878c54522621a94135dfe6322a9640f23e99e6ff4a1"} Oct 29 23:10:18.031954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.031927 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:18.171640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.171611 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-xks-ca\") pod \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " Oct 29 23:10:18.171793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.171671 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-openshift-ca\") pod \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " Oct 29 23:10:18.171793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.171776 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3d8c66-cdba-430d-8644-2c7a0a893f94-kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/5c3d8c66-cdba-430d-8644-2c7a0a893f94-kube-api-access-cqh8x\") pod \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\" (UID: \"5c3d8c66-cdba-430d-8644-2c7a0a893f94\") " Oct 29 23:10:18.172064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.172035 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-openshift-ca" pod "5c3d8c66-cdba-430d-8644-2c7a0a893f94" (UID: "5c3d8c66-cdba-430d-8644-2c7a0a893f94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:18.172464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.172446 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.173826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.173797 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-xks-ca" pod "5c3d8c66-cdba-430d-8644-2c7a0a893f94" (UID: "5c3d8c66-cdba-430d-8644-2c7a0a893f94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:18.174152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.174132 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3d8c66-cdba-430d-8644-2c7a0a893f94-kube-api-access-cqh8x" pod "5c3d8c66-cdba-430d-8644-2c7a0a893f94" (UID: "5c3d8c66-cdba-430d-8644-2c7a0a893f94"). InnerVolumeSpecName "kube-api-access-cqh8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:18.273700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.273631 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3d8c66-cdba-430d-8644-2c7a0a893f94-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.273700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.273659 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqh8x\" (UniqueName: \"kubernetes.io/projected/5c3d8c66-cdba-430d-8644-2c7a0a893f94-kube-api-access-cqh8x\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:18.908393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.908359 5736 generic.go:356] "Generic (PLEG): container finished" podID="51965f81-04ba-4a8b-9bea-ce56e06e7316" containerID="92012f169ee8d0a687c05760d7d4cfba41486c6e41249e014ee7d7c35363b470" exitCode=0 Oct 29 23:10:18.908828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.908436 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" event={"ID":"51965f81-04ba-4a8b-9bea-ce56e06e7316","Type":"ContainerDied","Data":"92012f169ee8d0a687c05760d7d4cfba41486c6e41249e014ee7d7c35363b470"} Oct 29 23:10:18.909951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.909930 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" Oct 29 23:10:18.910117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.909974 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7" event={"ID":"5c3d8c66-cdba-430d-8644-2c7a0a893f94","Type":"ContainerDied","Data":"2292e9247569d376e7e174dc06ce30b967e347314793f594b19ed80fb44ecf91"} Oct 29 23:10:18.910117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:18.910019 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2292e9247569d376e7e174dc06ce30b967e347314793f594b19ed80fb44ecf91" Oct 29 23:10:20.090692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.090662 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:20.208400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.208316 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-openshift-ca\") pod \"51965f81-04ba-4a8b-9bea-ce56e06e7316\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " Oct 29 23:10:20.208573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.208428 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-xks-ca\") pod \"51965f81-04ba-4a8b-9bea-ce56e06e7316\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " Oct 29 23:10:20.208573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.208450 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51965f81-04ba-4a8b-9bea-ce56e06e7316-kube-api-access-f2fsj\" (UniqueName: \"kubernetes.io/projected/51965f81-04ba-4a8b-9bea-ce56e06e7316-kube-api-access-f2fsj\") pod \"51965f81-04ba-4a8b-9bea-ce56e06e7316\" (UID: \"51965f81-04ba-4a8b-9bea-ce56e06e7316\") " Oct 29 23:10:20.208756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.208723 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-openshift-ca" pod "51965f81-04ba-4a8b-9bea-ce56e06e7316" (UID: "51965f81-04ba-4a8b-9bea-ce56e06e7316"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:20.209087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.209063 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.210498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.210470 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-xks-ca" pod "51965f81-04ba-4a8b-9bea-ce56e06e7316" (UID: "51965f81-04ba-4a8b-9bea-ce56e06e7316"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:20.210730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.210708 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51965f81-04ba-4a8b-9bea-ce56e06e7316-kube-api-access-f2fsj" pod "51965f81-04ba-4a8b-9bea-ce56e06e7316" (UID: "51965f81-04ba-4a8b-9bea-ce56e06e7316"). InnerVolumeSpecName "kube-api-access-f2fsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:20.310251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.310218 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51965f81-04ba-4a8b-9bea-ce56e06e7316-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.310251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.310254 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2fsj\" (UniqueName: \"kubernetes.io/projected/51965f81-04ba-4a8b-9bea-ce56e06e7316-kube-api-access-f2fsj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.930313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.930284 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" Oct 29 23:10:20.930313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.930291 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh" event={"ID":"51965f81-04ba-4a8b-9bea-ce56e06e7316","Type":"ContainerDied","Data":"071800635ee9d112a1f99878c54522621a94135dfe6322a9640f23e99e6ff4a1"} Oct 29 23:10:20.930528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:20.930340 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="071800635ee9d112a1f99878c54522621a94135dfe6322a9640f23e99e6ff4a1" Oct 29 23:10:24.267977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.267938 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85"] Oct 29 23:10:24.268564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.268547 5736 cpu_manager.go:414] "containerMap: removing container" podUID="51965f81-04ba-4a8b-9bea-ce56e06e7316" containerName="revoke-keys" Oct 29 23:10:24.268623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.268566 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51965f81-04ba-4a8b-9bea-ce56e06e7316" containerName="revoke-keys" Oct 29 23:10:24.268623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.268598 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5c3d8c66-cdba-430d-8644-2c7a0a893f94" containerName="revoke-keys" Oct 29 23:10:24.268623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.268606 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3d8c66-cdba-430d-8644-2c7a0a893f94" containerName="revoke-keys" Oct 29 23:10:24.268747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.268737 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="51965f81-04ba-4a8b-9bea-ce56e06e7316" containerName="revoke-keys" Oct 29 23:10:24.268780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.268752 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3d8c66-cdba-430d-8644-2c7a0a893f94" containerName="revoke-keys" Oct 29 23:10:24.272355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.272325 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.314394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.314366 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85"] Oct 29 23:10:24.374798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.374751 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.374977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.374832 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.374977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.374914 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.374977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.374956 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.375181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.374998 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.375181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.375076 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.375181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.375146 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.375332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.375307 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.375429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.375411 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2pr2\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-kube-api-access-q2pr2\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.476476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.476436 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2pr2\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-kube-api-access-q2pr2\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.476716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.476503 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.476716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.476557 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.476716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.476648 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.476716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.476687 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.476961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.476725 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.476961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.476803 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.476961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.476871 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.477115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.476968 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.477115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.477022 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.477244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.477140 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.477342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.477304 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.477397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.477353 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.477485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.477461 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.479132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.479094 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.479244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.479171 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.499868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.499838 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.499991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.499969 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2pr2\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-kube-api-access-q2pr2\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.629552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.629473 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:24.775286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.775176 5736 scope.go:122] "RemoveContainer" containerID="2b3af4fe5cc94ba701578d8a6a2a8608eea80bba8d3ece93119061631977e453" Oct 29 23:10:24.788368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.788350 5736 scope.go:122] "RemoveContainer" containerID="8aaf0818af9ce92cf0f163ec3da1aae1cee724a46979d345c05f18053812e77d" Oct 29 23:10:24.798475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.798455 5736 scope.go:122] "RemoveContainer" containerID="3df52e5aaef8e7398038d04e8a160d51e352cf1d2be8f31b40e486a8124007a6" Oct 29 23:10:24.813192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.813158 5736 scope.go:122] "RemoveContainer" containerID="3bc866bf4b15c48cc69f16e78275b6473cc44b01cc87e4f3101debc511016f0f" Oct 29 23:10:24.842686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.842547 5736 scope.go:122] "RemoveContainer" containerID="9346fa8b2a901a66574894c142dc73b7c4f1729fa4a545e99d8941c627a3c973" Oct 29 23:10:24.958836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.958668 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85"] Oct 29 23:10:24.962165 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:24.962140 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abd5efc_ca12_4692_bc81_e62dadfd60a7.slice/crio-4fc7fd361157283c8df7462cada6dc6716a0b57b381cb6756933c389b572ab96 WatchSource:0}: Error finding container 4fc7fd361157283c8df7462cada6dc6716a0b57b381cb6756933c389b572ab96: Status 404 returned error can't find the container with id 4fc7fd361157283c8df7462cada6dc6716a0b57b381cb6756933c389b572ab96 Oct 29 23:10:24.964981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.964950 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:24.965044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.965017 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:24.965090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:24.965047 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:25.975446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:25.975403 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" event={"ID":"1abd5efc-ca12-4692-bc81-e62dadfd60a7","Type":"ContainerStarted","Data":"92ea1a1f028c0140728146266dcd6ad2cb5164b6de819e76e2c3260bde8ca9b2"} Oct 29 23:10:25.975903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:25.975506 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" event={"ID":"1abd5efc-ca12-4692-bc81-e62dadfd60a7","Type":"ContainerStarted","Data":"4fc7fd361157283c8df7462cada6dc6716a0b57b381cb6756933c389b572ab96"} Oct 29 23:10:25.982129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:25.982102 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv"] Oct 29 23:10:25.986247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:25.986222 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.011645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.011615 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv"] Oct 29 23:10:26.029618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.029521 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" podStartSLOduration=2.029502619 podStartE2EDuration="2.029502619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:26.023114645 +0000 UTC m=+1623.139034255" watchObservedRunningTime="2026-10-29 23:10:26.029502619 +0000 UTC m=+1623.145422231" Oct 29 23:10:26.102262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.102225 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.102430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.102286 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.102545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.102516 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.102623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.102561 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.102670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.102647 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.102764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.102750 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.102846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.102826 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.102997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.102981 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnf6c\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-kube-api-access-xnf6c\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.103048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.103038 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.204863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.204823 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnf6c\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-kube-api-access-xnf6c\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.204891 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.204953 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205006 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205143 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205172 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205198 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205260 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205326 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205359 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205498 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205515 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205560 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.205860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.205840 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.207641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.207615 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.207769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.207753 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.228548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.228494 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnf6c\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-kube-api-access-xnf6c\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.228548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.228523 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.303768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.303737 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:26.630048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.629973 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:26.656534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.656502 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:10:26.656734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.656555 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 29 23:10:26.661512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.661485 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv"] Oct 29 23:10:26.665397 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:26.665369 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bbb4ac_5639_4e05_8260_6ef27d0e526c.slice/crio-f76b9eecf0f8f4af533ab708319b10017a3cf030f0edfb871d6db904c26980af WatchSource:0}: Error finding container f76b9eecf0f8f4af533ab708319b10017a3cf030f0edfb871d6db904c26980af: Status 404 returned error can't find the container with id f76b9eecf0f8f4af533ab708319b10017a3cf030f0edfb871d6db904c26980af Oct 29 23:10:26.668631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.668559 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:26.668728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.668662 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:26.668728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.668701 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:26.825498 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:10:26.825455 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:10:26.984515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.984476 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" event={"ID":"06bbb4ac-5639-4e05-8260-6ef27d0e526c","Type":"ContainerStarted","Data":"15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15"} Oct 29 23:10:26.985539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:26.984537 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" event={"ID":"06bbb4ac-5639-4e05-8260-6ef27d0e526c","Type":"ContainerStarted","Data":"f76b9eecf0f8f4af533ab708319b10017a3cf030f0edfb871d6db904c26980af"} Oct 29 23:10:27.038612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:27.031015 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podStartSLOduration=2.030997987 podStartE2EDuration="2.030997987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:27.02849532 +0000 UTC m=+1624.144414926" watchObservedRunningTime="2026-10-29 23:10:27.030997987 +0000 UTC m=+1624.146917594" Oct 29 23:10:27.304672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:27.304566 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:27.305994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:27.305964 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:10:27.306071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:27.306021 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 23:10:27.643712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:27.630843 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:10:27.643712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:27.630908 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 29 23:10:28.305011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:28.304981 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:10:28.305573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:28.305548 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 23:10:28.652262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:28.651705 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:10:28.652262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:28.651786 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 29 23:10:29.304362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:29.304324 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:10:29.304551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:29.304392 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 23:10:29.630718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:29.630328 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:10:29.630718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:29.630374 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 29 23:10:30.304373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:30.304337 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:10:30.304549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:30.304402 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 23:10:30.698035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:30.697941 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:10:30.698647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:30.698568 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 29 23:10:31.304684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:31.304652 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:10:31.304871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:31.304708 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 23:10:31.630818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:31.630726 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:10:31.631083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:31.631055 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 29 23:10:32.304421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:32.304376 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:10:32.304856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:32.304449 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 23:10:33.108444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.108410 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd"] Oct 29 23:10:33.128659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.128617 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.184722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.171552 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.215847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.215778 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.216749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.216154 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.217039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.217016 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.217222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.217206 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.217352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.217338 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.217461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.217448 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e8d379-c210-485a-85ee-f1c2d00cad24-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.217636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.217620 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.217778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.217764 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b42f\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-kube-api-access-8b42f\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.219198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.219133 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd"] Oct 29 23:10:33.287602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.281250 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:33.287602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.281798 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:33.287602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.283019 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:10:33.304257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.304221 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:10:33.304444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.304270 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 23:10:33.319060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319029 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319107 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319164 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319200 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319223 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319247 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e8d379-c210-485a-85ee-f1c2d00cad24-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319304 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319335 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b42f\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-kube-api-access-8b42f\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319436 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319566 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319610 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319848 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.319929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319869 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.320055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.319983 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e8d379-c210-485a-85ee-f1c2d00cad24-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.321848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.321825 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.322203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.322185 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.342508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.342481 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.342832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.342805 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b42f\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-kube-api-access-8b42f\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.461128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.461096 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:33.852100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.852062 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd"] Oct 29 23:10:33.852943 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:33.852916 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e8d379_c210_485a_85ee_f1c2d00cad24.slice/crio-f6dd5c5e68383683cb14f79a963f098eaa21f0fc11a92ed83c782ffb6949da36 WatchSource:0}: Error finding container f6dd5c5e68383683cb14f79a963f098eaa21f0fc11a92ed83c782ffb6949da36: Status 404 returned error can't find the container with id f6dd5c5e68383683cb14f79a963f098eaa21f0fc11a92ed83c782ffb6949da36 Oct 29 23:10:33.855454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.855420 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:33.855550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.855482 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:33.855550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:33.855511 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:10:34.196374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.196282 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt"] Oct 29 23:10:34.200254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.200226 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d21fd-fz2pt"] Oct 29 23:10:34.232804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.232770 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" event={"ID":"47e8d379-c210-485a-85ee-f1c2d00cad24","Type":"ContainerStarted","Data":"7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f"} Oct 29 23:10:34.232964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.232831 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" event={"ID":"47e8d379-c210-485a-85ee-f1c2d00cad24","Type":"ContainerStarted","Data":"f6dd5c5e68383683cb14f79a963f098eaa21f0fc11a92ed83c782ffb6949da36"} Oct 29 23:10:34.280719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.280663 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" podStartSLOduration=1.28064423 podStartE2EDuration="1.28064423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:34.276834819 +0000 UTC m=+1631.392754427" watchObservedRunningTime="2026-10-29 23:10:34.28064423 +0000 UTC m=+1631.396563837" Oct 29 23:10:34.304956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.304925 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:10:34.305429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.304982 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 23:10:34.464915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.464829 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:34.476656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.475943 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:10:34.476656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:34.476012 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 23:10:35.006699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.006661 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c728b3ea-1482-4a9e-8582-f8ea6a448f14" path="/var/lib/kubelet/pods/c728b3ea-1482-4a9e-8582-f8ea6a448f14/volumes" Oct 29 23:10:35.308957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.308883 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:35.309409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.309301 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:35.309966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.309947 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:10:35.464025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.463985 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:10:35.464346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.464311 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 23:10:35.471353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.471313 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq"] Oct 29 23:10:35.478692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.478669 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.494097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.494073 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:10:35.494871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.494277 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:10:35.496798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.496769 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq"] Oct 29 23:10:35.574042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.573946 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5"] Oct 29 23:10:35.578304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.578270 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.598021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.597995 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5"] Oct 29 23:10:35.672829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.672797 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c748663d-s9msq\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.673002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.672833 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57df55e1-cnmz5\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.673002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.672858 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57df55e1-cnmz5\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.673099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.673017 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz6tq\" (UniqueName: \"kubernetes.io/projected/2b418bcd-7812-4d03-9d5e-3581982db4d2-kube-api-access-cz6tq\") pod \"maas-api-revoke-sub-c748663d-s9msq\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.673099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.673051 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c748663d-s9msq\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.673099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.673075 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cdrs\" (UniqueName: \"kubernetes.io/projected/1ec983e3-0680-4148-ba8e-393cb06599f9-kube-api-access-6cdrs\") pod \"maas-api-revoke-sub-57df55e1-cnmz5\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.774256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.774220 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c748663d-s9msq\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.774441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.774270 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57df55e1-cnmz5\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.774441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.774301 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57df55e1-cnmz5\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.774441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.774418 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz6tq\" (UniqueName: \"kubernetes.io/projected/2b418bcd-7812-4d03-9d5e-3581982db4d2-kube-api-access-cz6tq\") pod \"maas-api-revoke-sub-c748663d-s9msq\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.774626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.774463 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c748663d-s9msq\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.774626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.774506 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cdrs\" (UniqueName: \"kubernetes.io/projected/1ec983e3-0680-4148-ba8e-393cb06599f9-kube-api-access-6cdrs\") pod \"maas-api-revoke-sub-57df55e1-cnmz5\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.775034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.775010 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c748663d-s9msq\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.775230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.775207 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57df55e1-cnmz5\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.778305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.777374 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c748663d-s9msq\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.778886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.778839 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57df55e1-cnmz5\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.797930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.797901 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cdrs\" (UniqueName: \"kubernetes.io/projected/1ec983e3-0680-4148-ba8e-393cb06599f9-kube-api-access-6cdrs\") pod \"maas-api-revoke-sub-57df55e1-cnmz5\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.798092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.798069 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz6tq\" (UniqueName: \"kubernetes.io/projected/2b418bcd-7812-4d03-9d5e-3581982db4d2-kube-api-access-cz6tq\") pod \"maas-api-revoke-sub-c748663d-s9msq\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.810048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.809912 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:35.896492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.894299 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:35.974345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.972955 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c"] Oct 29 23:10:35.980692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.980663 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:35.996822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:35.996339 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c"] Oct 29 23:10:36.082724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.082635 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912c4231-8553-491e-8716-536551e89ded-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a29365cb-q9t8c\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.082904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.082838 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912c4231-8553-491e-8716-536551e89ded-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a29365cb-q9t8c\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.083025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.083006 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjhs\" (UniqueName: \"kubernetes.io/projected/912c4231-8553-491e-8716-536551e89ded-kube-api-access-qzjhs\") pod \"maas-api-revoke-sub-a29365cb-q9t8c\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.162421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.162331 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq"] Oct 29 23:10:36.166306 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:36.166278 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b418bcd_7812_4d03_9d5e_3581982db4d2.slice/crio-21a13fafaffb263864c46107279d491f5463fb5494a44db2fa4b2777c2bc657f WatchSource:0}: Error finding container 21a13fafaffb263864c46107279d491f5463fb5494a44db2fa4b2777c2bc657f: Status 404 returned error can't find the container with id 21a13fafaffb263864c46107279d491f5463fb5494a44db2fa4b2777c2bc657f Oct 29 23:10:36.184647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.184627 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912c4231-8553-491e-8716-536551e89ded-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a29365cb-q9t8c\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.184731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.184717 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912c4231-8553-491e-8716-536551e89ded-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a29365cb-q9t8c\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.184821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.184803 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjhs\" (UniqueName: \"kubernetes.io/projected/912c4231-8553-491e-8716-536551e89ded-kube-api-access-qzjhs\") pod \"maas-api-revoke-sub-a29365cb-q9t8c\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.185315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.185292 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912c4231-8553-491e-8716-536551e89ded-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a29365cb-q9t8c\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.187082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.187059 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912c4231-8553-491e-8716-536551e89ded-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a29365cb-q9t8c\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.207804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.207782 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjhs\" (UniqueName: \"kubernetes.io/projected/912c4231-8553-491e-8716-536551e89ded-kube-api-access-qzjhs\") pod \"maas-api-revoke-sub-a29365cb-q9t8c\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.259946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.259329 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5"] Oct 29 23:10:36.263267 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:36.263244 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec983e3_0680_4148_ba8e_393cb06599f9.slice/crio-800aebdf8920008f53aac59f899e328f126091c2b75d9834db8ff5aad9bcf62a WatchSource:0}: Error finding container 800aebdf8920008f53aac59f899e328f126091c2b75d9834db8ff5aad9bcf62a: Status 404 returned error can't find the container with id 800aebdf8920008f53aac59f899e328f126091c2b75d9834db8ff5aad9bcf62a Oct 29 23:10:36.271454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.271425 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" event={"ID":"2b418bcd-7812-4d03-9d5e-3581982db4d2","Type":"ContainerStarted","Data":"21a13fafaffb263864c46107279d491f5463fb5494a44db2fa4b2777c2bc657f"} Oct 29 23:10:36.273064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.273026 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" event={"ID":"1ec983e3-0680-4148-ba8e-393cb06599f9","Type":"ContainerStarted","Data":"800aebdf8920008f53aac59f899e328f126091c2b75d9834db8ff5aad9bcf62a"} Oct 29 23:10:36.320668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.320645 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:36.462655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.462620 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:10:36.462803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.462672 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 23:10:36.761544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:36.761509 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c"] Oct 29 23:10:36.762370 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:36.762344 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912c4231_8553_491e_8716_536551e89ded.slice/crio-d109d741e094a560b5af413331b6f6344a452bac32582a7fe131a5e1d28e56e2 WatchSource:0}: Error finding container d109d741e094a560b5af413331b6f6344a452bac32582a7fe131a5e1d28e56e2: Status 404 returned error can't find the container with id d109d741e094a560b5af413331b6f6344a452bac32582a7fe131a5e1d28e56e2 Oct 29 23:10:37.310380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:37.310344 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" event={"ID":"1ec983e3-0680-4148-ba8e-393cb06599f9","Type":"ContainerStarted","Data":"0919ad9e51dab8d6243ec48b7a2cf1de533ac08ea2a801180aecefe5e5ed49e4"} Oct 29 23:10:37.312454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:37.312406 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" event={"ID":"912c4231-8553-491e-8716-536551e89ded","Type":"ContainerStarted","Data":"3284d06e79e8279f4726b307b4548cb3b7246b81d57002a559f6ca2eb85aa5fe"} Oct 29 23:10:37.312669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:37.312471 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" event={"ID":"912c4231-8553-491e-8716-536551e89ded","Type":"ContainerStarted","Data":"d109d741e094a560b5af413331b6f6344a452bac32582a7fe131a5e1d28e56e2"} Oct 29 23:10:37.314926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:37.314900 5736 generic.go:356] "Generic (PLEG): container finished" podID="2b418bcd-7812-4d03-9d5e-3581982db4d2" containerID="18ce5ddfceb5f3dac47f396f1453bb50a03eea87f5f00a033dc293b5fbe445d7" exitCode=0 Oct 29 23:10:37.315042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:37.314944 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" event={"ID":"2b418bcd-7812-4d03-9d5e-3581982db4d2","Type":"ContainerDied","Data":"18ce5ddfceb5f3dac47f396f1453bb50a03eea87f5f00a033dc293b5fbe445d7"} Oct 29 23:10:37.413115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:37.413039 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" podStartSLOduration=2.413019453 podStartE2EDuration="2.413019453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:37.369805051 +0000 UTC m=+1634.485724682" watchObservedRunningTime="2026-10-29 23:10:37.413019453 +0000 UTC m=+1634.528939060" Oct 29 23:10:37.461679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:37.458601 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" podStartSLOduration=2.458568421 podStartE2EDuration="2.458568421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:37.455987287 +0000 UTC m=+1634.571906906" watchObservedRunningTime="2026-10-29 23:10:37.458568421 +0000 UTC m=+1634.574488046" Oct 29 23:10:37.465285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:37.465251 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:10:37.465427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:37.465300 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 23:10:38.323558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.323522 5736 generic.go:356] "Generic (PLEG): container finished" podID="1ec983e3-0680-4148-ba8e-393cb06599f9" containerID="0919ad9e51dab8d6243ec48b7a2cf1de533ac08ea2a801180aecefe5e5ed49e4" exitCode=0 Oct 29 23:10:38.323761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.323608 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" event={"ID":"1ec983e3-0680-4148-ba8e-393cb06599f9","Type":"ContainerDied","Data":"0919ad9e51dab8d6243ec48b7a2cf1de533ac08ea2a801180aecefe5e5ed49e4"} Oct 29 23:10:38.325266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.325240 5736 generic.go:356] "Generic (PLEG): container finished" podID="912c4231-8553-491e-8716-536551e89ded" containerID="3284d06e79e8279f4726b307b4548cb3b7246b81d57002a559f6ca2eb85aa5fe" exitCode=0 Oct 29 23:10:38.325389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.325318 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" event={"ID":"912c4231-8553-491e-8716-536551e89ded","Type":"ContainerDied","Data":"3284d06e79e8279f4726b307b4548cb3b7246b81d57002a559f6ca2eb85aa5fe"} Oct 29 23:10:38.462090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.462045 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:10:38.462639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.462111 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 23:10:38.565822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.565792 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:38.629930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.629861 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b418bcd-7812-4d03-9d5e-3581982db4d2-kube-api-access-cz6tq\" (UniqueName: \"kubernetes.io/projected/2b418bcd-7812-4d03-9d5e-3581982db4d2-kube-api-access-cz6tq\") pod \"2b418bcd-7812-4d03-9d5e-3581982db4d2\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " Oct 29 23:10:38.630072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.630006 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-openshift-ca\") pod \"2b418bcd-7812-4d03-9d5e-3581982db4d2\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " Oct 29 23:10:38.630072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.630034 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-xks-ca\") pod \"2b418bcd-7812-4d03-9d5e-3581982db4d2\" (UID: \"2b418bcd-7812-4d03-9d5e-3581982db4d2\") " Oct 29 23:10:38.630396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.630370 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-openshift-ca" pod "2b418bcd-7812-4d03-9d5e-3581982db4d2" (UID: "2b418bcd-7812-4d03-9d5e-3581982db4d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:38.653602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.630644 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:38.657072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.657038 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-xks-ca" pod "2b418bcd-7812-4d03-9d5e-3581982db4d2" (UID: "2b418bcd-7812-4d03-9d5e-3581982db4d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:38.657515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.657478 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b418bcd-7812-4d03-9d5e-3581982db4d2-kube-api-access-cz6tq" pod "2b418bcd-7812-4d03-9d5e-3581982db4d2" (UID: "2b418bcd-7812-4d03-9d5e-3581982db4d2"). InnerVolumeSpecName "kube-api-access-cz6tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:38.755630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.743785 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b418bcd-7812-4d03-9d5e-3581982db4d2-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:38.755800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:38.755670 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz6tq\" (UniqueName: \"kubernetes.io/projected/2b418bcd-7812-4d03-9d5e-3581982db4d2-kube-api-access-cz6tq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:39.349661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.347810 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" Oct 29 23:10:39.349661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.348652 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq" event={"ID":"2b418bcd-7812-4d03-9d5e-3581982db4d2","Type":"ContainerDied","Data":"21a13fafaffb263864c46107279d491f5463fb5494a44db2fa4b2777c2bc657f"} Oct 29 23:10:39.349661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.348706 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a13fafaffb263864c46107279d491f5463fb5494a44db2fa4b2777c2bc657f" Oct 29 23:10:39.462193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.462154 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:10:39.462723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.462212 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 23:10:39.606684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.599606 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:39.680676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.680648 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:39.691312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.691277 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-xks-ca\") pod \"1ec983e3-0680-4148-ba8e-393cb06599f9\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " Oct 29 23:10:39.691459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.691420 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-openshift-ca\") pod \"1ec983e3-0680-4148-ba8e-393cb06599f9\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " Oct 29 23:10:39.692594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.691680 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec983e3-0680-4148-ba8e-393cb06599f9-kube-api-access-6cdrs\" (UniqueName: \"kubernetes.io/projected/1ec983e3-0680-4148-ba8e-393cb06599f9-kube-api-access-6cdrs\") pod \"1ec983e3-0680-4148-ba8e-393cb06599f9\" (UID: \"1ec983e3-0680-4148-ba8e-393cb06599f9\") " Oct 29 23:10:39.692594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.692102 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-openshift-ca" pod "1ec983e3-0680-4148-ba8e-393cb06599f9" (UID: "1ec983e3-0680-4148-ba8e-393cb06599f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:39.692594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.692355 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:39.694874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.694823 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec983e3-0680-4148-ba8e-393cb06599f9-kube-api-access-6cdrs" pod "1ec983e3-0680-4148-ba8e-393cb06599f9" (UID: "1ec983e3-0680-4148-ba8e-393cb06599f9"). InnerVolumeSpecName "kube-api-access-6cdrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:39.695126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.695094 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-xks-ca" pod "1ec983e3-0680-4148-ba8e-393cb06599f9" (UID: "1ec983e3-0680-4148-ba8e-393cb06599f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:39.793433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.793397 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912c4231-8553-491e-8716-536551e89ded-kube-api-access-qzjhs\" (UniqueName: \"kubernetes.io/projected/912c4231-8553-491e-8716-536551e89ded-kube-api-access-qzjhs\") pod \"912c4231-8553-491e-8716-536551e89ded\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " Oct 29 23:10:39.793433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.793438 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912c4231-8553-491e-8716-536551e89ded-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912c4231-8553-491e-8716-536551e89ded-maas-api-openshift-ca\") pod \"912c4231-8553-491e-8716-536551e89ded\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " Oct 29 23:10:39.793807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.793469 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912c4231-8553-491e-8716-536551e89ded-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912c4231-8553-491e-8716-536551e89ded-maas-api-xks-ca\") pod \"912c4231-8553-491e-8716-536551e89ded\" (UID: \"912c4231-8553-491e-8716-536551e89ded\") " Oct 29 23:10:39.793878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.793832 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912c4231-8553-491e-8716-536551e89ded-maas-api-openshift-ca" pod "912c4231-8553-491e-8716-536551e89ded" (UID: "912c4231-8553-491e-8716-536551e89ded"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:39.794292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.794267 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912c4231-8553-491e-8716-536551e89ded-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:39.794292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.794292 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cdrs\" (UniqueName: \"kubernetes.io/projected/1ec983e3-0680-4148-ba8e-393cb06599f9-kube-api-access-6cdrs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:39.794499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.794302 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec983e3-0680-4148-ba8e-393cb06599f9-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:39.795670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.795649 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912c4231-8553-491e-8716-536551e89ded-maas-api-xks-ca" pod "912c4231-8553-491e-8716-536551e89ded" (UID: "912c4231-8553-491e-8716-536551e89ded"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:39.795743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.795722 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912c4231-8553-491e-8716-536551e89ded-kube-api-access-qzjhs" pod "912c4231-8553-491e-8716-536551e89ded" (UID: "912c4231-8553-491e-8716-536551e89ded"). InnerVolumeSpecName "kube-api-access-qzjhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:39.895342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.895269 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzjhs\" (UniqueName: \"kubernetes.io/projected/912c4231-8553-491e-8716-536551e89ded-kube-api-access-qzjhs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:39.895342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:39.895295 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912c4231-8553-491e-8716-536551e89ded-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:40.355712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:40.355674 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" event={"ID":"1ec983e3-0680-4148-ba8e-393cb06599f9","Type":"ContainerDied","Data":"800aebdf8920008f53aac59f899e328f126091c2b75d9834db8ff5aad9bcf62a"} Oct 29 23:10:40.355893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:40.355736 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="800aebdf8920008f53aac59f899e328f126091c2b75d9834db8ff5aad9bcf62a" Oct 29 23:10:40.355893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:40.355686 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5" Oct 29 23:10:40.360393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:40.360372 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" Oct 29 23:10:40.360763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:40.360424 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c" event={"ID":"912c4231-8553-491e-8716-536551e89ded","Type":"ContainerDied","Data":"d109d741e094a560b5af413331b6f6344a452bac32582a7fe131a5e1d28e56e2"} Oct 29 23:10:40.360763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:40.360474 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d109d741e094a560b5af413331b6f6344a452bac32582a7fe131a5e1d28e56e2" Oct 29 23:10:40.461903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:40.461864 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:10:40.462046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:40.461946 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 23:10:41.466176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:41.466146 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:41.466712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:41.466689 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:41.467297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:41.467278 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:10:46.405837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.405802 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4"] Oct 29 23:10:46.406376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.406361 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1ec983e3-0680-4148-ba8e-393cb06599f9" containerName="revoke-keys" Oct 29 23:10:46.406431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.406377 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec983e3-0680-4148-ba8e-393cb06599f9" containerName="revoke-keys" Oct 29 23:10:46.406431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.406403 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2b418bcd-7812-4d03-9d5e-3581982db4d2" containerName="revoke-keys" Oct 29 23:10:46.406431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.406409 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b418bcd-7812-4d03-9d5e-3581982db4d2" containerName="revoke-keys" Oct 29 23:10:46.406431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.406417 5736 cpu_manager.go:414] "containerMap: removing container" podUID="912c4231-8553-491e-8716-536551e89ded" containerName="revoke-keys" Oct 29 23:10:46.406431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.406423 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912c4231-8553-491e-8716-536551e89ded" containerName="revoke-keys" Oct 29 23:10:46.406650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.406506 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b418bcd-7812-4d03-9d5e-3581982db4d2" containerName="revoke-keys" Oct 29 23:10:46.406650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.406514 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="912c4231-8553-491e-8716-536551e89ded" containerName="revoke-keys" Oct 29 23:10:46.406650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.406526 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec983e3-0680-4148-ba8e-393cb06599f9" containerName="revoke-keys" Oct 29 23:10:46.409980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.409964 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.425748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.425723 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:10:46.438081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.438055 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:10:46.443328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.443305 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4"] Oct 29 23:10:46.490689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.490663 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6pr\" (UniqueName: \"kubernetes.io/projected/93b71ee4-4a70-49d7-960d-54d8ab1a2360-kube-api-access-rl6pr\") pod \"maas-api-revoke-sub-745c9def-wk8z4\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.490811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.490744 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-745c9def-wk8z4\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.490983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.490965 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-745c9def-wk8z4\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.592316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.592289 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6pr\" (UniqueName: \"kubernetes.io/projected/93b71ee4-4a70-49d7-960d-54d8ab1a2360-kube-api-access-rl6pr\") pod \"maas-api-revoke-sub-745c9def-wk8z4\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.592445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.592319 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-745c9def-wk8z4\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.592445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.592424 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-745c9def-wk8z4\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.593171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.593144 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-745c9def-wk8z4\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.594714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.594690 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-745c9def-wk8z4\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.615336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.615314 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6pr\" (UniqueName: \"kubernetes.io/projected/93b71ee4-4a70-49d7-960d-54d8ab1a2360-kube-api-access-rl6pr\") pod \"maas-api-revoke-sub-745c9def-wk8z4\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:46.724433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:46.724408 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:47.027925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.027899 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4"] Oct 29 23:10:47.031067 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:47.031037 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b71ee4_4a70_49d7_960d_54d8ab1a2360.slice/crio-5689d3512e419cfae7bf6a13e0b3fb0093b63b7e5393de44cacd62faee8e908c WatchSource:0}: Error finding container 5689d3512e419cfae7bf6a13e0b3fb0093b63b7e5393de44cacd62faee8e908c: Status 404 returned error can't find the container with id 5689d3512e419cfae7bf6a13e0b3fb0093b63b7e5393de44cacd62faee8e908c Oct 29 23:10:47.057570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.057542 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5"] Oct 29 23:10:47.062883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.062863 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.092515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.092487 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5"] Oct 29 23:10:47.201347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.201318 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cggn7\" (UniqueName: \"kubernetes.io/projected/8412b23f-0352-4d3f-9bf6-05a0b9985942-kube-api-access-cggn7\") pod \"maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.201509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.201384 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.201650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.201620 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.298513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.298454 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps"] Oct 29 23:10:47.302626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.302608 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.303057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.303038 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cggn7\" (UniqueName: \"kubernetes.io/projected/8412b23f-0352-4d3f-9bf6-05a0b9985942-kube-api-access-cggn7\") pod \"maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.303117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.303106 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.303249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.303236 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.303852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.303832 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.321617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.321591 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps"] Oct 29 23:10:47.331325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.331295 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.331728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.331704 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cggn7\" (UniqueName: \"kubernetes.io/projected/8412b23f-0352-4d3f-9bf6-05a0b9985942-kube-api-access-cggn7\") pod \"maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.380676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.380643 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:47.404193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.404165 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qzsc\" (UniqueName: \"kubernetes.io/projected/6af3095e-02e5-4b19-9a36-22d595f9b64f-kube-api-access-5qzsc\") pod \"maas-api-revoke-sub-29072a62-ltsps\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.404354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.404339 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29072a62-ltsps\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.404456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.404443 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29072a62-ltsps\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.442553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.442456 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" event={"ID":"93b71ee4-4a70-49d7-960d-54d8ab1a2360","Type":"ContainerStarted","Data":"5689d3512e419cfae7bf6a13e0b3fb0093b63b7e5393de44cacd62faee8e908c"} Oct 29 23:10:47.506043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.506015 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qzsc\" (UniqueName: \"kubernetes.io/projected/6af3095e-02e5-4b19-9a36-22d595f9b64f-kube-api-access-5qzsc\") pod \"maas-api-revoke-sub-29072a62-ltsps\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.506183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.506167 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29072a62-ltsps\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.506291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.506273 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29072a62-ltsps\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.506862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.506836 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29072a62-ltsps\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.508485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.508457 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29072a62-ltsps\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.528820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.528797 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qzsc\" (UniqueName: \"kubernetes.io/projected/6af3095e-02e5-4b19-9a36-22d595f9b64f-kube-api-access-5qzsc\") pod \"maas-api-revoke-sub-29072a62-ltsps\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.616738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.616679 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:47.698843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.697952 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5"] Oct 29 23:10:47.926034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:47.925952 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps"] Oct 29 23:10:47.928817 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:47.928790 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af3095e_02e5_4b19_9a36_22d595f9b64f.slice/crio-380deed7cf77cdf1a6b76cb20342aa22468df3205b0c452050fbf5ba3bdc11bd WatchSource:0}: Error finding container 380deed7cf77cdf1a6b76cb20342aa22468df3205b0c452050fbf5ba3bdc11bd: Status 404 returned error can't find the container with id 380deed7cf77cdf1a6b76cb20342aa22468df3205b0c452050fbf5ba3bdc11bd Oct 29 23:10:48.451656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:48.451607 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" event={"ID":"6af3095e-02e5-4b19-9a36-22d595f9b64f","Type":"ContainerStarted","Data":"74108cb99f3358a5cf9ec8ad20ca79a24259689b8df5af78b8ba8ffa64cea6ff"} Oct 29 23:10:48.452076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:48.451674 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" event={"ID":"6af3095e-02e5-4b19-9a36-22d595f9b64f","Type":"ContainerStarted","Data":"380deed7cf77cdf1a6b76cb20342aa22468df3205b0c452050fbf5ba3bdc11bd"} Oct 29 23:10:48.453368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:48.453340 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" event={"ID":"8412b23f-0352-4d3f-9bf6-05a0b9985942","Type":"ContainerStarted","Data":"f5967c2218914f848041dedfcc35655f4ee3e18620cfcec1a673ed65fbd4263e"} Oct 29 23:10:48.453499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:48.453385 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" event={"ID":"8412b23f-0352-4d3f-9bf6-05a0b9985942","Type":"ContainerStarted","Data":"9efa536bd4167af89dc5b91a3a0cf88840b81e227c28dbecf9c8b0cac818daaa"} Oct 29 23:10:48.454954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:48.454930 5736 generic.go:356] "Generic (PLEG): container finished" podID="93b71ee4-4a70-49d7-960d-54d8ab1a2360" containerID="4b7c89e14364fdac084c2a411667f10458c370eeeaf5db1b2edf68a350b21084" exitCode=0 Oct 29 23:10:48.455042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:48.454963 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" event={"ID":"93b71ee4-4a70-49d7-960d-54d8ab1a2360","Type":"ContainerDied","Data":"4b7c89e14364fdac084c2a411667f10458c370eeeaf5db1b2edf68a350b21084"} Oct 29 23:10:48.493608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:48.493547 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" podStartSLOduration=1.493534308 podStartE2EDuration="1.493534308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:48.491199524 +0000 UTC m=+1645.607119130" watchObservedRunningTime="2026-10-29 23:10:48.493534308 +0000 UTC m=+1645.609453916" Oct 29 23:10:48.573294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:48.573241 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" podStartSLOduration=1.573225571 podStartE2EDuration="1.573225571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:48.569473276 +0000 UTC m=+1645.685392881" watchObservedRunningTime="2026-10-29 23:10:48.573225571 +0000 UTC m=+1645.689145177" Oct 29 23:10:49.463276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.463239 5736 generic.go:356] "Generic (PLEG): container finished" podID="6af3095e-02e5-4b19-9a36-22d595f9b64f" containerID="74108cb99f3358a5cf9ec8ad20ca79a24259689b8df5af78b8ba8ffa64cea6ff" exitCode=0 Oct 29 23:10:49.463710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.463324 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" event={"ID":"6af3095e-02e5-4b19-9a36-22d595f9b64f","Type":"ContainerDied","Data":"74108cb99f3358a5cf9ec8ad20ca79a24259689b8df5af78b8ba8ffa64cea6ff"} Oct 29 23:10:49.465012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.464988 5736 generic.go:356] "Generic (PLEG): container finished" podID="8412b23f-0352-4d3f-9bf6-05a0b9985942" containerID="f5967c2218914f848041dedfcc35655f4ee3e18620cfcec1a673ed65fbd4263e" exitCode=0 Oct 29 23:10:49.465146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.465072 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" event={"ID":"8412b23f-0352-4d3f-9bf6-05a0b9985942","Type":"ContainerDied","Data":"f5967c2218914f848041dedfcc35655f4ee3e18620cfcec1a673ed65fbd4263e"} Oct 29 23:10:49.621686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.621662 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:49.738486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.738402 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-openshift-ca\") pod \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " Oct 29 23:10:49.738674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.738540 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-xks-ca\") pod \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " Oct 29 23:10:49.738674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.738565 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b71ee4-4a70-49d7-960d-54d8ab1a2360-kube-api-access-rl6pr\" (UniqueName: \"kubernetes.io/projected/93b71ee4-4a70-49d7-960d-54d8ab1a2360-kube-api-access-rl6pr\") pod \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\" (UID: \"93b71ee4-4a70-49d7-960d-54d8ab1a2360\") " Oct 29 23:10:49.738863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.738838 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-openshift-ca" pod "93b71ee4-4a70-49d7-960d-54d8ab1a2360" (UID: "93b71ee4-4a70-49d7-960d-54d8ab1a2360"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:49.739228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.739207 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:49.740724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.740704 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-xks-ca" pod "93b71ee4-4a70-49d7-960d-54d8ab1a2360" (UID: "93b71ee4-4a70-49d7-960d-54d8ab1a2360"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:49.740811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.740784 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b71ee4-4a70-49d7-960d-54d8ab1a2360-kube-api-access-rl6pr" pod "93b71ee4-4a70-49d7-960d-54d8ab1a2360" (UID: "93b71ee4-4a70-49d7-960d-54d8ab1a2360"). InnerVolumeSpecName "kube-api-access-rl6pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:49.841299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.841258 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b71ee4-4a70-49d7-960d-54d8ab1a2360-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:49.841299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:49.841300 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl6pr\" (UniqueName: \"kubernetes.io/projected/93b71ee4-4a70-49d7-960d-54d8ab1a2360-kube-api-access-rl6pr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:50.404602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.391632 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg"] Oct 29 23:10:50.404602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.392884 5736 cpu_manager.go:414] "containerMap: removing container" podUID="93b71ee4-4a70-49d7-960d-54d8ab1a2360" containerName="revoke-keys" Oct 29 23:10:50.404602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.392907 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b71ee4-4a70-49d7-960d-54d8ab1a2360" containerName="revoke-keys" Oct 29 23:10:50.404602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.393051 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b71ee4-4a70-49d7-960d-54d8ab1a2360" containerName="revoke-keys" Oct 29 23:10:50.404602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.397731 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.423422 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg"] Oct 29 23:10:50.465328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.465293 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a29c5bfc-mz7zg\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.466936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.465381 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpmp\" (UniqueName: \"kubernetes.io/projected/363e8920-b210-4a9c-86fb-c9049b44c0e6-kube-api-access-fqpmp\") pod \"maas-api-revoke-sub-a29c5bfc-mz7zg\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.466936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.465520 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a29c5bfc-mz7zg\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.530857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.530813 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" event={"ID":"93b71ee4-4a70-49d7-960d-54d8ab1a2360","Type":"ContainerDied","Data":"5689d3512e419cfae7bf6a13e0b3fb0093b63b7e5393de44cacd62faee8e908c"} Oct 29 23:10:50.530857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.530847 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4" Oct 29 23:10:50.531087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.530913 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5689d3512e419cfae7bf6a13e0b3fb0093b63b7e5393de44cacd62faee8e908c" Oct 29 23:10:50.630268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.629407 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a29c5bfc-mz7zg\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.630268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.629610 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a29c5bfc-mz7zg\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.630268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.629645 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpmp\" (UniqueName: \"kubernetes.io/projected/363e8920-b210-4a9c-86fb-c9049b44c0e6-kube-api-access-fqpmp\") pod \"maas-api-revoke-sub-a29c5bfc-mz7zg\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.630544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.630428 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a29c5bfc-mz7zg\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.632630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.632432 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a29c5bfc-mz7zg\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.655330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.655236 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpmp\" (UniqueName: \"kubernetes.io/projected/363e8920-b210-4a9c-86fb-c9049b44c0e6-kube-api-access-fqpmp\") pod \"maas-api-revoke-sub-a29c5bfc-mz7zg\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.752103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.752077 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:50.757374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.757354 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:50.759559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.759536 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:50.832809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.832775 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-openshift-ca\") pod \"8412b23f-0352-4d3f-9bf6-05a0b9985942\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " Oct 29 23:10:50.832928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.832875 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-xks-ca\") pod \"6af3095e-02e5-4b19-9a36-22d595f9b64f\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " Oct 29 23:10:50.832982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.832952 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-openshift-ca\") pod \"6af3095e-02e5-4b19-9a36-22d595f9b64f\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " Oct 29 23:10:50.833147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.833127 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af3095e-02e5-4b19-9a36-22d595f9b64f-kube-api-access-5qzsc\" (UniqueName: \"kubernetes.io/projected/6af3095e-02e5-4b19-9a36-22d595f9b64f-kube-api-access-5qzsc\") pod \"6af3095e-02e5-4b19-9a36-22d595f9b64f\" (UID: \"6af3095e-02e5-4b19-9a36-22d595f9b64f\") " Oct 29 23:10:50.833205 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.833176 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8412b23f-0352-4d3f-9bf6-05a0b9985942-kube-api-access-cggn7\" (UniqueName: \"kubernetes.io/projected/8412b23f-0352-4d3f-9bf6-05a0b9985942-kube-api-access-cggn7\") pod \"8412b23f-0352-4d3f-9bf6-05a0b9985942\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " Oct 29 23:10:50.833295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.833273 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-xks-ca\") pod \"8412b23f-0352-4d3f-9bf6-05a0b9985942\" (UID: \"8412b23f-0352-4d3f-9bf6-05a0b9985942\") " Oct 29 23:10:50.833570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.833544 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-openshift-ca" pod "8412b23f-0352-4d3f-9bf6-05a0b9985942" (UID: "8412b23f-0352-4d3f-9bf6-05a0b9985942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:50.834023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.833999 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-openshift-ca" pod "6af3095e-02e5-4b19-9a36-22d595f9b64f" (UID: "6af3095e-02e5-4b19-9a36-22d595f9b64f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:50.834188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.834167 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:50.836618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.836465 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-xks-ca" pod "8412b23f-0352-4d3f-9bf6-05a0b9985942" (UID: "8412b23f-0352-4d3f-9bf6-05a0b9985942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:50.836618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.836560 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8412b23f-0352-4d3f-9bf6-05a0b9985942-kube-api-access-cggn7" pod "8412b23f-0352-4d3f-9bf6-05a0b9985942" (UID: "8412b23f-0352-4d3f-9bf6-05a0b9985942"). InnerVolumeSpecName "kube-api-access-cggn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:50.836747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.836672 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-xks-ca" pod "6af3095e-02e5-4b19-9a36-22d595f9b64f" (UID: "6af3095e-02e5-4b19-9a36-22d595f9b64f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:50.836747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.836702 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af3095e-02e5-4b19-9a36-22d595f9b64f-kube-api-access-5qzsc" pod "6af3095e-02e5-4b19-9a36-22d595f9b64f" (UID: "6af3095e-02e5-4b19-9a36-22d595f9b64f"). InnerVolumeSpecName "kube-api-access-5qzsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:50.898076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.898046 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2"] Oct 29 23:10:50.899000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.898978 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6af3095e-02e5-4b19-9a36-22d595f9b64f" containerName="revoke-keys" Oct 29 23:10:50.899064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.899003 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af3095e-02e5-4b19-9a36-22d595f9b64f" containerName="revoke-keys" Oct 29 23:10:50.899103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.899063 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8412b23f-0352-4d3f-9bf6-05a0b9985942" containerName="revoke-keys" Oct 29 23:10:50.899103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.899072 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8412b23f-0352-4d3f-9bf6-05a0b9985942" containerName="revoke-keys" Oct 29 23:10:50.899252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.899237 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8412b23f-0352-4d3f-9bf6-05a0b9985942" containerName="revoke-keys" Oct 29 23:10:50.899284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.899265 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af3095e-02e5-4b19-9a36-22d595f9b64f" containerName="revoke-keys" Oct 29 23:10:50.904944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.904920 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:50.924911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.922728 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2"] Oct 29 23:10:50.935948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.935920 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:50.936065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.935954 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af3095e-02e5-4b19-9a36-22d595f9b64f-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:50.936065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.935970 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qzsc\" (UniqueName: \"kubernetes.io/projected/6af3095e-02e5-4b19-9a36-22d595f9b64f-kube-api-access-5qzsc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:50.936065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.935986 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cggn7\" (UniqueName: \"kubernetes.io/projected/8412b23f-0352-4d3f-9bf6-05a0b9985942-kube-api-access-cggn7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:50.936065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:50.935999 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8412b23f-0352-4d3f-9bf6-05a0b9985942-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:51.037262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.037231 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b239048-l6sj2\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.037448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.037430 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b239048-l6sj2\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.037550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.037529 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drfrk\" (UniqueName: \"kubernetes.io/projected/c1e80024-ec48-4ecf-bfab-6f6898768b5d-kube-api-access-drfrk\") pod \"maas-api-revoke-sub-8b239048-l6sj2\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.063605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.063566 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg"] Oct 29 23:10:51.066783 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:51.066753 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363e8920_b210_4a9c_86fb_c9049b44c0e6.slice/crio-69f5840c5960f632866ea447a7a52d198a5b13de4af20310f836cd9988a04a00 WatchSource:0}: Error finding container 69f5840c5960f632866ea447a7a52d198a5b13de4af20310f836cd9988a04a00: Status 404 returned error can't find the container with id 69f5840c5960f632866ea447a7a52d198a5b13de4af20310f836cd9988a04a00 Oct 29 23:10:51.083674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.083645 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh"] Oct 29 23:10:51.087746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.087722 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-921680-hpvjh"] Oct 29 23:10:51.139444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.139394 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b239048-l6sj2\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.139665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.139639 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b239048-l6sj2\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.139747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.139720 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drfrk\" (UniqueName: \"kubernetes.io/projected/c1e80024-ec48-4ecf-bfab-6f6898768b5d-kube-api-access-drfrk\") pod \"maas-api-revoke-sub-8b239048-l6sj2\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.140278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.140245 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b239048-l6sj2\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.143869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.143842 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b239048-l6sj2\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.164134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.164067 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drfrk\" (UniqueName: \"kubernetes.io/projected/c1e80024-ec48-4ecf-bfab-6f6898768b5d-kube-api-access-drfrk\") pod \"maas-api-revoke-sub-8b239048-l6sj2\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.229926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.229897 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:51.540909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.540880 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" Oct 29 23:10:51.541327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.540915 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps" event={"ID":"6af3095e-02e5-4b19-9a36-22d595f9b64f","Type":"ContainerDied","Data":"380deed7cf77cdf1a6b76cb20342aa22468df3205b0c452050fbf5ba3bdc11bd"} Oct 29 23:10:51.541327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.540965 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="380deed7cf77cdf1a6b76cb20342aa22468df3205b0c452050fbf5ba3bdc11bd" Oct 29 23:10:51.542892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.542871 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" Oct 29 23:10:51.543045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.542930 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5" event={"ID":"8412b23f-0352-4d3f-9bf6-05a0b9985942","Type":"ContainerDied","Data":"9efa536bd4167af89dc5b91a3a0cf88840b81e227c28dbecf9c8b0cac818daaa"} Oct 29 23:10:51.543045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.542976 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9efa536bd4167af89dc5b91a3a0cf88840b81e227c28dbecf9c8b0cac818daaa" Oct 29 23:10:51.545868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.545844 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" event={"ID":"363e8920-b210-4a9c-86fb-c9049b44c0e6","Type":"ContainerStarted","Data":"53105424a63f9fe62b5b22e1cb4448b77ab342859329563094debf533ba448f1"} Oct 29 23:10:51.545982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.545886 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" event={"ID":"363e8920-b210-4a9c-86fb-c9049b44c0e6","Type":"ContainerStarted","Data":"69f5840c5960f632866ea447a7a52d198a5b13de4af20310f836cd9988a04a00"} Oct 29 23:10:51.549945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.549925 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2"] Oct 29 23:10:51.551818 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:51.551790 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e80024_ec48_4ecf_bfab_6f6898768b5d.slice/crio-9df49816295384d39d51360f31bcb6b99ebe07fd24705b615705a17d9edaed2e WatchSource:0}: Error finding container 9df49816295384d39d51360f31bcb6b99ebe07fd24705b615705a17d9edaed2e: Status 404 returned error can't find the container with id 9df49816295384d39d51360f31bcb6b99ebe07fd24705b615705a17d9edaed2e Oct 29 23:10:51.591748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:51.591706 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" podStartSLOduration=1.591692855 podStartE2EDuration="1.591692855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:51.588646385 +0000 UTC m=+1648.704565988" watchObservedRunningTime="2026-10-29 23:10:51.591692855 +0000 UTC m=+1648.707612461" Oct 29 23:10:52.554893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:52.554857 5736 generic.go:356] "Generic (PLEG): container finished" podID="c1e80024-ec48-4ecf-bfab-6f6898768b5d" containerID="5ea7e520697e2a9f9f0221a2f7bbdc2a651b052ed7aa369f14aa87ef0a326269" exitCode=0 Oct 29 23:10:52.555331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:52.554939 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" event={"ID":"c1e80024-ec48-4ecf-bfab-6f6898768b5d","Type":"ContainerDied","Data":"5ea7e520697e2a9f9f0221a2f7bbdc2a651b052ed7aa369f14aa87ef0a326269"} Oct 29 23:10:52.555331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:52.554991 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" event={"ID":"c1e80024-ec48-4ecf-bfab-6f6898768b5d","Type":"ContainerStarted","Data":"9df49816295384d39d51360f31bcb6b99ebe07fd24705b615705a17d9edaed2e"} Oct 29 23:10:52.556637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:52.556615 5736 generic.go:356] "Generic (PLEG): container finished" podID="363e8920-b210-4a9c-86fb-c9049b44c0e6" containerID="53105424a63f9fe62b5b22e1cb4448b77ab342859329563094debf533ba448f1" exitCode=0 Oct 29 23:10:52.556740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:52.556645 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" event={"ID":"363e8920-b210-4a9c-86fb-c9049b44c0e6","Type":"ContainerDied","Data":"53105424a63f9fe62b5b22e1cb4448b77ab342859329563094debf533ba448f1"} Oct 29 23:10:53.006460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.006424 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51965f81-04ba-4a8b-9bea-ce56e06e7316" path="/var/lib/kubelet/pods/51965f81-04ba-4a8b-9bea-ce56e06e7316/volumes" Oct 29 23:10:53.744746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.744719 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:53.769463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.769437 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:53.896146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.896076 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-openshift-ca\") pod \"363e8920-b210-4a9c-86fb-c9049b44c0e6\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " Oct 29 23:10:53.896146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.896126 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e80024-ec48-4ecf-bfab-6f6898768b5d-kube-api-access-drfrk\" (UniqueName: \"kubernetes.io/projected/c1e80024-ec48-4ecf-bfab-6f6898768b5d-kube-api-access-drfrk\") pod \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " Oct 29 23:10:53.896363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.896222 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-openshift-ca\") pod \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " Oct 29 23:10:53.896363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.896254 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363e8920-b210-4a9c-86fb-c9049b44c0e6-kube-api-access-fqpmp\" (UniqueName: \"kubernetes.io/projected/363e8920-b210-4a9c-86fb-c9049b44c0e6-kube-api-access-fqpmp\") pod \"363e8920-b210-4a9c-86fb-c9049b44c0e6\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " Oct 29 23:10:53.896363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.896311 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-xks-ca\") pod \"363e8920-b210-4a9c-86fb-c9049b44c0e6\" (UID: \"363e8920-b210-4a9c-86fb-c9049b44c0e6\") " Oct 29 23:10:53.896522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.896476 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-xks-ca\") pod \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\" (UID: \"c1e80024-ec48-4ecf-bfab-6f6898768b5d\") " Oct 29 23:10:53.896522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.896486 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-openshift-ca" pod "363e8920-b210-4a9c-86fb-c9049b44c0e6" (UID: "363e8920-b210-4a9c-86fb-c9049b44c0e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:53.896679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.896644 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-openshift-ca" pod "c1e80024-ec48-4ecf-bfab-6f6898768b5d" (UID: "c1e80024-ec48-4ecf-bfab-6f6898768b5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:53.897332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.897304 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:53.897471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.897335 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:53.898696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.898663 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e80024-ec48-4ecf-bfab-6f6898768b5d-kube-api-access-drfrk" pod "c1e80024-ec48-4ecf-bfab-6f6898768b5d" (UID: "c1e80024-ec48-4ecf-bfab-6f6898768b5d"). InnerVolumeSpecName "kube-api-access-drfrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:53.898780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.898726 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-xks-ca" pod "363e8920-b210-4a9c-86fb-c9049b44c0e6" (UID: "363e8920-b210-4a9c-86fb-c9049b44c0e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:53.898780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.898744 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363e8920-b210-4a9c-86fb-c9049b44c0e6-kube-api-access-fqpmp" pod "363e8920-b210-4a9c-86fb-c9049b44c0e6" (UID: "363e8920-b210-4a9c-86fb-c9049b44c0e6"). InnerVolumeSpecName "kube-api-access-fqpmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:53.898943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.898920 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-xks-ca" pod "c1e80024-ec48-4ecf-bfab-6f6898768b5d" (UID: "c1e80024-ec48-4ecf-bfab-6f6898768b5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:10:53.999007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.998981 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1e80024-ec48-4ecf-bfab-6f6898768b5d-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:53.999007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.999004 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drfrk\" (UniqueName: \"kubernetes.io/projected/c1e80024-ec48-4ecf-bfab-6f6898768b5d-kube-api-access-drfrk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:53.999156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.999014 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpmp\" (UniqueName: \"kubernetes.io/projected/363e8920-b210-4a9c-86fb-c9049b44c0e6-kube-api-access-fqpmp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:53.999156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:53.999025 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e8920-b210-4a9c-86fb-c9049b44c0e6-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:10:54.572867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:54.572832 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" Oct 29 23:10:54.573030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:54.572856 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2" event={"ID":"c1e80024-ec48-4ecf-bfab-6f6898768b5d","Type":"ContainerDied","Data":"9df49816295384d39d51360f31bcb6b99ebe07fd24705b615705a17d9edaed2e"} Oct 29 23:10:54.573030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:54.572917 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9df49816295384d39d51360f31bcb6b99ebe07fd24705b615705a17d9edaed2e" Oct 29 23:10:54.574348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:54.574328 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" Oct 29 23:10:54.574439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:54.574360 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg" event={"ID":"363e8920-b210-4a9c-86fb-c9049b44c0e6","Type":"ContainerDied","Data":"69f5840c5960f632866ea447a7a52d198a5b13de4af20310f836cd9988a04a00"} Oct 29 23:10:54.574439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:54.574401 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69f5840c5960f632866ea447a7a52d198a5b13de4af20310f836cd9988a04a00" Oct 29 23:10:56.824895 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:10:56.824853 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:10:58.485737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:58.485699 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x"] Oct 29 23:10:58.486996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:58.486560 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" containerID="cri-o://0e84d65f4008c2a67da589e991e2108879ad494c7e48d2d1ff744df76a70d084" gracePeriod=30 Oct 29 23:10:59.154539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.154502 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m"] Oct 29 23:10:59.155130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.155111 5736 cpu_manager.go:414] "containerMap: removing container" podUID="363e8920-b210-4a9c-86fb-c9049b44c0e6" containerName="revoke-keys" Oct 29 23:10:59.155221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.155129 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363e8920-b210-4a9c-86fb-c9049b44c0e6" containerName="revoke-keys" Oct 29 23:10:59.155221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.155145 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c1e80024-ec48-4ecf-bfab-6f6898768b5d" containerName="revoke-keys" Oct 29 23:10:59.155221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.155151 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e80024-ec48-4ecf-bfab-6f6898768b5d" containerName="revoke-keys" Oct 29 23:10:59.155378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.155292 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="363e8920-b210-4a9c-86fb-c9049b44c0e6" containerName="revoke-keys" Oct 29 23:10:59.155378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.155302 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e80024-ec48-4ecf-bfab-6f6898768b5d" containerName="revoke-keys" Oct 29 23:10:59.160058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.160037 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.178123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.178101 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m"] Oct 29 23:10:59.186718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.186687 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:10:59.186843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.186800 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:10:59.214098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.214040 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.214293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.214242 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.214478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.214354 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn8gx\" (UniqueName: \"kubernetes.io/projected/3d679de3-e7cd-4fca-9951-3a26ace94acf-kube-api-access-fn8gx\") pod \"maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.315402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.315372 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.315628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.315475 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.315628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.315526 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn8gx\" (UniqueName: \"kubernetes.io/projected/3d679de3-e7cd-4fca-9951-3a26ace94acf-kube-api-access-fn8gx\") pod \"maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.316071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.316045 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.317863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.317842 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.338836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.338809 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn8gx\" (UniqueName: \"kubernetes.io/projected/3d679de3-e7cd-4fca-9951-3a26ace94acf-kube-api-access-fn8gx\") pod \"maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.474636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.474606 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:10:59.802783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.802756 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m"] Oct 29 23:10:59.805617 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:10:59.805559 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d679de3_e7cd_4fca_9951_3a26ace94acf.slice/crio-34bb67869c49aad001caba8f15f9d3ecd2b4bb978482d76f5b2a537413fe5fd2 WatchSource:0}: Error finding container 34bb67869c49aad001caba8f15f9d3ecd2b4bb978482d76f5b2a537413fe5fd2: Status 404 returned error can't find the container with id 34bb67869c49aad001caba8f15f9d3ecd2b4bb978482d76f5b2a537413fe5fd2 Oct 29 23:10:59.814702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.814675 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d"] Oct 29 23:10:59.821333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.821313 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:10:59.839447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.839419 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d"] Oct 29 23:10:59.923258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.923227 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqsh2\" (UniqueName: \"kubernetes.io/projected/913373b1-8a1b-466a-8f5b-1a365257f8b2-kube-api-access-gqsh2\") pod \"maas-api-revoke-sub-c7d30db0-9cp8d\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:10:59.923434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.923353 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d30db0-9cp8d\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:10:59.923507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:10:59.923490 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d30db0-9cp8d\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:00.024676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.024638 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d30db0-9cp8d\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:00.024887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.024778 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqsh2\" (UniqueName: \"kubernetes.io/projected/913373b1-8a1b-466a-8f5b-1a365257f8b2-kube-api-access-gqsh2\") pod \"maas-api-revoke-sub-c7d30db0-9cp8d\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:00.024887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.024849 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d30db0-9cp8d\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:00.025385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.025352 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d30db0-9cp8d\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:00.027211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.027187 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d30db0-9cp8d\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:00.048305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.048280 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqsh2\" (UniqueName: \"kubernetes.io/projected/913373b1-8a1b-466a-8f5b-1a365257f8b2-kube-api-access-gqsh2\") pod \"maas-api-revoke-sub-c7d30db0-9cp8d\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:00.198395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.198357 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:00.501404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.501375 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d"] Oct 29 23:11:00.504414 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:00.504386 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913373b1_8a1b_466a_8f5b_1a365257f8b2.slice/crio-9e2c5528c981240c49a770c1ced3a9819b786b80a9a4b2bf372b1cb8411aa289 WatchSource:0}: Error finding container 9e2c5528c981240c49a770c1ced3a9819b786b80a9a4b2bf372b1cb8411aa289: Status 404 returned error can't find the container with id 9e2c5528c981240c49a770c1ced3a9819b786b80a9a4b2bf372b1cb8411aa289 Oct 29 23:11:00.638858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.638822 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" event={"ID":"913373b1-8a1b-466a-8f5b-1a365257f8b2","Type":"ContainerStarted","Data":"9e2c5528c981240c49a770c1ced3a9819b786b80a9a4b2bf372b1cb8411aa289"} Oct 29 23:11:00.640648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.640620 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" event={"ID":"3d679de3-e7cd-4fca-9951-3a26ace94acf","Type":"ContainerStarted","Data":"a16dca94633f7eabe3b61478f9d47ad9a71f937115b6137b4ac41627fe100d05"} Oct 29 23:11:00.640782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.640674 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" event={"ID":"3d679de3-e7cd-4fca-9951-3a26ace94acf","Type":"ContainerStarted","Data":"34bb67869c49aad001caba8f15f9d3ecd2b4bb978482d76f5b2a537413fe5fd2"} Oct 29 23:11:00.682439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:00.682394 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" podStartSLOduration=1.6823808279999999 podStartE2EDuration="1.682380828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:00.680542423 +0000 UTC m=+1657.796462029" watchObservedRunningTime="2026-10-29 23:11:00.682380828 +0000 UTC m=+1657.798300431" Oct 29 23:11:01.651929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:01.651839 5736 generic.go:356] "Generic (PLEG): container finished" podID="913373b1-8a1b-466a-8f5b-1a365257f8b2" containerID="c9aeefac7bb620dad9750387f66c9f9041239780397fae883582c233b670d95e" exitCode=0 Oct 29 23:11:01.652386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:01.651922 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" event={"ID":"913373b1-8a1b-466a-8f5b-1a365257f8b2","Type":"ContainerDied","Data":"c9aeefac7bb620dad9750387f66c9f9041239780397fae883582c233b670d95e"} Oct 29 23:11:01.653981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:01.653956 5736 generic.go:356] "Generic (PLEG): container finished" podID="3d679de3-e7cd-4fca-9951-3a26ace94acf" containerID="a16dca94633f7eabe3b61478f9d47ad9a71f937115b6137b4ac41627fe100d05" exitCode=0 Oct 29 23:11:01.654100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:01.654044 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" event={"ID":"3d679de3-e7cd-4fca-9951-3a26ace94acf","Type":"ContainerDied","Data":"a16dca94633f7eabe3b61478f9d47ad9a71f937115b6137b4ac41627fe100d05"} Oct 29 23:11:02.845468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.845434 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:02.879097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.879064 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:11:02.968276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.968245 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913373b1-8a1b-466a-8f5b-1a365257f8b2-kube-api-access-gqsh2\" (UniqueName: \"kubernetes.io/projected/913373b1-8a1b-466a-8f5b-1a365257f8b2-kube-api-access-gqsh2\") pod \"913373b1-8a1b-466a-8f5b-1a365257f8b2\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " Oct 29 23:11:02.968469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.968329 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-xks-ca\") pod \"913373b1-8a1b-466a-8f5b-1a365257f8b2\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " Oct 29 23:11:02.968469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.968435 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d679de3-e7cd-4fca-9951-3a26ace94acf-kube-api-access-fn8gx\" (UniqueName: \"kubernetes.io/projected/3d679de3-e7cd-4fca-9951-3a26ace94acf-kube-api-access-fn8gx\") pod \"3d679de3-e7cd-4fca-9951-3a26ace94acf\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " Oct 29 23:11:02.968610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.968494 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-openshift-ca\") pod \"913373b1-8a1b-466a-8f5b-1a365257f8b2\" (UID: \"913373b1-8a1b-466a-8f5b-1a365257f8b2\") " Oct 29 23:11:02.968672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.968634 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-xks-ca\") pod \"3d679de3-e7cd-4fca-9951-3a26ace94acf\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " Oct 29 23:11:02.968776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.968760 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-openshift-ca\") pod \"3d679de3-e7cd-4fca-9951-3a26ace94acf\" (UID: \"3d679de3-e7cd-4fca-9951-3a26ace94acf\") " Oct 29 23:11:02.969133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.969092 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-openshift-ca" pod "913373b1-8a1b-466a-8f5b-1a365257f8b2" (UID: "913373b1-8a1b-466a-8f5b-1a365257f8b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:02.969242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.969185 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-openshift-ca" pod "3d679de3-e7cd-4fca-9951-3a26ace94acf" (UID: "3d679de3-e7cd-4fca-9951-3a26ace94acf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:02.969439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.969419 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:02.969531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.969442 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:02.970753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.970725 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913373b1-8a1b-466a-8f5b-1a365257f8b2-kube-api-access-gqsh2" pod "913373b1-8a1b-466a-8f5b-1a365257f8b2" (UID: "913373b1-8a1b-466a-8f5b-1a365257f8b2"). InnerVolumeSpecName "kube-api-access-gqsh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:02.970876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.970783 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d679de3-e7cd-4fca-9951-3a26ace94acf-kube-api-access-fn8gx" pod "3d679de3-e7cd-4fca-9951-3a26ace94acf" (UID: "3d679de3-e7cd-4fca-9951-3a26ace94acf"). InnerVolumeSpecName "kube-api-access-fn8gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:02.971011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.970991 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-xks-ca" pod "3d679de3-e7cd-4fca-9951-3a26ace94acf" (UID: "3d679de3-e7cd-4fca-9951-3a26ace94acf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:02.971289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:02.971271 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-xks-ca" pod "913373b1-8a1b-466a-8f5b-1a365257f8b2" (UID: "913373b1-8a1b-466a-8f5b-1a365257f8b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:03.071142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.071110 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn8gx\" (UniqueName: \"kubernetes.io/projected/3d679de3-e7cd-4fca-9951-3a26ace94acf-kube-api-access-fn8gx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.071142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.071141 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d679de3-e7cd-4fca-9951-3a26ace94acf-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.071344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.071153 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqsh2\" (UniqueName: \"kubernetes.io/projected/913373b1-8a1b-466a-8f5b-1a365257f8b2-kube-api-access-gqsh2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.071344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.071163 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913373b1-8a1b-466a-8f5b-1a365257f8b2-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.676222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.675612 5736 generic.go:356] "Generic (PLEG): container finished" podID="f79fd910-a277-4b22-849e-bead58762990" containerID="0e84d65f4008c2a67da589e991e2108879ad494c7e48d2d1ff744df76a70d084" exitCode=0 Oct 29 23:11:03.676222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.675839 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" event={"ID":"f79fd910-a277-4b22-849e-bead58762990","Type":"ContainerDied","Data":"0e84d65f4008c2a67da589e991e2108879ad494c7e48d2d1ff744df76a70d084"} Oct 29 23:11:03.676222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.675929 5736 scope.go:122] "RemoveContainer" containerID="cb80dd9eb8bec558e57f778e4f30a14b216e33c397a2f790ed61b84f3e297f24" Oct 29 23:11:03.680038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.680005 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" Oct 29 23:11:03.680038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.680034 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d" event={"ID":"913373b1-8a1b-466a-8f5b-1a365257f8b2","Type":"ContainerDied","Data":"9e2c5528c981240c49a770c1ced3a9819b786b80a9a4b2bf372b1cb8411aa289"} Oct 29 23:11:03.680388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.680081 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e2c5528c981240c49a770c1ced3a9819b786b80a9a4b2bf372b1cb8411aa289" Oct 29 23:11:03.687978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.687950 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" event={"ID":"3d679de3-e7cd-4fca-9951-3a26ace94acf","Type":"ContainerDied","Data":"34bb67869c49aad001caba8f15f9d3ecd2b4bb978482d76f5b2a537413fe5fd2"} Oct 29 23:11:03.688124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.688000 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34bb67869c49aad001caba8f15f9d3ecd2b4bb978482d76f5b2a537413fe5fd2" Oct 29 23:11:03.688124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.688065 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m" Oct 29 23:11:03.804049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.804025 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:11:03.886344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886307 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-istio-token\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-istio-token\") pod \"f79fd910-a277-4b22-849e-bead58762990\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " Oct 29 23:11:03.886862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886382 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-certs\") pod \"f79fd910-a277-4b22-849e-bead58762990\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " Oct 29 23:11:03.886862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886504 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79fd910-a277-4b22-849e-bead58762990-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79fd910-a277-4b22-849e-bead58762990-istiod-ca-cert\") pod \"f79fd910-a277-4b22-849e-bead58762990\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " Oct 29 23:11:03.886862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886535 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-data\") pod \"f79fd910-a277-4b22-849e-bead58762990\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " Oct 29 23:11:03.886862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886571 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-kube-api-access-4bkq7\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-kube-api-access-4bkq7\") pod \"f79fd910-a277-4b22-849e-bead58762990\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " Oct 29 23:11:03.886862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886630 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-credential-socket\") pod \"f79fd910-a277-4b22-849e-bead58762990\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " Oct 29 23:11:03.886862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886693 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f79fd910-a277-4b22-849e-bead58762990-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79fd910-a277-4b22-849e-bead58762990-istio-podinfo\") pod \"f79fd910-a277-4b22-849e-bead58762990\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " Oct 29 23:11:03.886862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886774 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-socket\") pod \"f79fd910-a277-4b22-849e-bead58762990\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " Oct 29 23:11:03.886862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886791 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-certs" pod "f79fd910-a277-4b22-849e-bead58762990" (UID: "f79fd910-a277-4b22-849e-bead58762990"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:03.886862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886817 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-envoy\") pod \"f79fd910-a277-4b22-849e-bead58762990\" (UID: \"f79fd910-a277-4b22-849e-bead58762990\") " Oct 29 23:11:03.887392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.886931 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79fd910-a277-4b22-849e-bead58762990-istiod-ca-cert" pod "f79fd910-a277-4b22-849e-bead58762990" (UID: "f79fd910-a277-4b22-849e-bead58762990"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:03.887392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.887189 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-credential-socket" pod "f79fd910-a277-4b22-849e-bead58762990" (UID: "f79fd910-a277-4b22-849e-bead58762990"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:03.887392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.887368 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-data" pod "f79fd910-a277-4b22-849e-bead58762990" (UID: "f79fd910-a277-4b22-849e-bead58762990"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:03.887392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.887384 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-socket" pod "f79fd910-a277-4b22-849e-bead58762990" (UID: "f79fd910-a277-4b22-849e-bead58762990"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:03.887967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.887942 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.888056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.887975 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79fd910-a277-4b22-849e-bead58762990-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.888056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.887993 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.888056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.888009 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.888056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.888025 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.889410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.889292 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-kube-api-access-4bkq7" pod "f79fd910-a277-4b22-849e-bead58762990" (UID: "f79fd910-a277-4b22-849e-bead58762990"). InnerVolumeSpecName "kube-api-access-4bkq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:03.889748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.889728 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-istio-token" pod "f79fd910-a277-4b22-849e-bead58762990" (UID: "f79fd910-a277-4b22-849e-bead58762990"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:03.889844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.889775 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-envoy" pod "f79fd910-a277-4b22-849e-bead58762990" (UID: "f79fd910-a277-4b22-849e-bead58762990"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:03.891083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.891062 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f79fd910-a277-4b22-849e-bead58762990-istio-podinfo" pod "f79fd910-a277-4b22-849e-bead58762990" (UID: "f79fd910-a277-4b22-849e-bead58762990"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:03.989831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.989783 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bkq7\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-kube-api-access-4bkq7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.989831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.989826 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79fd910-a277-4b22-849e-bead58762990-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.989831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.989835 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79fd910-a277-4b22-849e-bead58762990-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:03.990077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:03.989846 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79fd910-a277-4b22-849e-bead58762990-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:04.697658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:04.697629 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" Oct 29 23:11:04.697658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:04.697640 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x" event={"ID":"f79fd910-a277-4b22-849e-bead58762990","Type":"ContainerDied","Data":"470dbd9da645e03c3d10e48b1907f83ff6808f9190619ffa7e5401c6b55b1be2"} Oct 29 23:11:04.697853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:04.697701 5736 scope.go:122] "RemoveContainer" containerID="0e84d65f4008c2a67da589e991e2108879ad494c7e48d2d1ff744df76a70d084" Oct 29 23:11:04.764499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:04.764471 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x"] Oct 29 23:11:04.776393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:04.776366 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-921680-openshift-default-64d78bd855-s8v6x"] Oct 29 23:11:05.004967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:05.004937 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79fd910-a277-4b22-849e-bead58762990" path="/var/lib/kubelet/pods/f79fd910-a277-4b22-849e-bead58762990/volumes" Oct 29 23:11:09.184753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.184723 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr"] Oct 29 23:11:09.185325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185304 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" Oct 29 23:11:09.185325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185325 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" Oct 29 23:11:09.185516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185363 5736 cpu_manager.go:414] "containerMap: removing container" podUID="913373b1-8a1b-466a-8f5b-1a365257f8b2" containerName="revoke-keys" Oct 29 23:11:09.185516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185371 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913373b1-8a1b-466a-8f5b-1a365257f8b2" containerName="revoke-keys" Oct 29 23:11:09.185516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185385 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3d679de3-e7cd-4fca-9951-3a26ace94acf" containerName="revoke-keys" Oct 29 23:11:09.185516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185394 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d679de3-e7cd-4fca-9951-3a26ace94acf" containerName="revoke-keys" Oct 29 23:11:09.185516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185427 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" Oct 29 23:11:09.185516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185435 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" Oct 29 23:11:09.185764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185593 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" Oct 29 23:11:09.185764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185604 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="913373b1-8a1b-466a-8f5b-1a365257f8b2" containerName="revoke-keys" Oct 29 23:11:09.185764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185612 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d679de3-e7cd-4fca-9951-3a26ace94acf" containerName="revoke-keys" Oct 29 23:11:09.185764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.185621 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79fd910-a277-4b22-849e-bead58762990" containerName="istio-proxy" Oct 29 23:11:09.190711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.190693 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.208263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.208242 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr"] Oct 29 23:11:09.216815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.216794 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:11:09.217067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.217040 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:11:09.268229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.268190 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4jd\" (UniqueName: \"kubernetes.io/projected/d460e938-bd5b-42e4-ad89-f6d8a75e35df-kube-api-access-8c4jd\") pod \"maas-api-revoke-sub-0f215939-f9wvr\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.268407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.268339 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f215939-f9wvr\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.268463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.268447 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f215939-f9wvr\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.370068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.370035 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f215939-f9wvr\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.370271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.370130 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4jd\" (UniqueName: \"kubernetes.io/projected/d460e938-bd5b-42e4-ad89-f6d8a75e35df-kube-api-access-8c4jd\") pod \"maas-api-revoke-sub-0f215939-f9wvr\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.370271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.370258 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f215939-f9wvr\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.370793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.370763 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f215939-f9wvr\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.372630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.372612 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f215939-f9wvr\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.393443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.393412 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4jd\" (UniqueName: \"kubernetes.io/projected/d460e938-bd5b-42e4-ad89-f6d8a75e35df-kube-api-access-8c4jd\") pod \"maas-api-revoke-sub-0f215939-f9wvr\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.504516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.504490 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:09.806779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:09.806755 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr"] Oct 29 23:11:09.809013 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:09.808985 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd460e938_bd5b_42e4_ad89_f6d8a75e35df.slice/crio-2cfe4e392b1d8810a28270c367b4f4bc44a1de4cea2a15532ea908480132b024 WatchSource:0}: Error finding container 2cfe4e392b1d8810a28270c367b4f4bc44a1de4cea2a15532ea908480132b024: Status 404 returned error can't find the container with id 2cfe4e392b1d8810a28270c367b4f4bc44a1de4cea2a15532ea908480132b024 Oct 29 23:11:10.188793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.188759 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg"] Oct 29 23:11:10.193022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.193001 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.213019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.212983 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg"] Oct 29 23:11:10.283703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.283672 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqvvb\" (UniqueName: \"kubernetes.io/projected/0304ee61-d364-4f68-afe6-04af71b0e1df-kube-api-access-zqvvb\") pod \"maas-api-revoke-sub-de852370-llflg\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.283872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.283797 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de852370-llflg\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.283872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.283854 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de852370-llflg\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.385010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.384980 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de852370-llflg\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.385178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.385158 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvvb\" (UniqueName: \"kubernetes.io/projected/0304ee61-d364-4f68-afe6-04af71b0e1df-kube-api-access-zqvvb\") pod \"maas-api-revoke-sub-de852370-llflg\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.385260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.385247 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de852370-llflg\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.385877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.385856 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de852370-llflg\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.387302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.387282 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de852370-llflg\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.408375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.408320 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqvvb\" (UniqueName: \"kubernetes.io/projected/0304ee61-d364-4f68-afe6-04af71b0e1df-kube-api-access-zqvvb\") pod \"maas-api-revoke-sub-de852370-llflg\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.506389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.506358 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:10.760501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.760333 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" event={"ID":"d460e938-bd5b-42e4-ad89-f6d8a75e35df","Type":"ContainerStarted","Data":"01e7081e9a68f220bc0de16242194ae135f73dc223130a698885d594ea5932e6"} Oct 29 23:11:10.760685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.760519 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" event={"ID":"d460e938-bd5b-42e4-ad89-f6d8a75e35df","Type":"ContainerStarted","Data":"2cfe4e392b1d8810a28270c367b4f4bc44a1de4cea2a15532ea908480132b024"} Oct 29 23:11:10.809223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.809130 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" podStartSLOduration=1.8091036379999998 podStartE2EDuration="1.809103638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:10.806004392 +0000 UTC m=+1667.921924011" watchObservedRunningTime="2026-10-29 23:11:10.809103638 +0000 UTC m=+1667.925023248" Oct 29 23:11:10.854410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:10.854375 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg"] Oct 29 23:11:11.561383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:11.561340 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5"] Oct 29 23:11:11.566257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:11.566230 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85c702-zfnf5"] Oct 29 23:11:11.771645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:11.769295 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" event={"ID":"0304ee61-d364-4f68-afe6-04af71b0e1df","Type":"ContainerStarted","Data":"69fe2b5850205513e0323cb25237fc3b33f8e00ccb5fa7035374863db141e335"} Oct 29 23:11:11.771645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:11.769376 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" event={"ID":"0304ee61-d364-4f68-afe6-04af71b0e1df","Type":"ContainerStarted","Data":"acd310e4a0d27fc4e1d2951da7611773ad31e455b25ebc5c8612e7f331ba306e"} Oct 29 23:11:11.771645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:11.771244 5736 generic.go:356] "Generic (PLEG): container finished" podID="d460e938-bd5b-42e4-ad89-f6d8a75e35df" containerID="01e7081e9a68f220bc0de16242194ae135f73dc223130a698885d594ea5932e6" exitCode=0 Oct 29 23:11:11.771645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:11.771364 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" event={"ID":"d460e938-bd5b-42e4-ad89-f6d8a75e35df","Type":"ContainerDied","Data":"01e7081e9a68f220bc0de16242194ae135f73dc223130a698885d594ea5932e6"} Oct 29 23:11:11.813338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:11.813294 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" podStartSLOduration=1.8132807340000001 podStartE2EDuration="1.813280734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:11.810948989 +0000 UTC m=+1668.926868596" watchObservedRunningTime="2026-10-29 23:11:11.813280734 +0000 UTC m=+1668.929200341" Oct 29 23:11:12.781327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:12.781291 5736 generic.go:356] "Generic (PLEG): container finished" podID="0304ee61-d364-4f68-afe6-04af71b0e1df" containerID="69fe2b5850205513e0323cb25237fc3b33f8e00ccb5fa7035374863db141e335" exitCode=0 Oct 29 23:11:12.781722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:12.781345 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" event={"ID":"0304ee61-d364-4f68-afe6-04af71b0e1df","Type":"ContainerDied","Data":"69fe2b5850205513e0323cb25237fc3b33f8e00ccb5fa7035374863db141e335"} Oct 29 23:11:12.975194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:12.975165 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:13.004221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.004190 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8412b23f-0352-4d3f-9bf6-05a0b9985942" path="/var/lib/kubelet/pods/8412b23f-0352-4d3f-9bf6-05a0b9985942/volumes" Oct 29 23:11:13.081197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.081101 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-xks-ca\") pod \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " Oct 29 23:11:13.081368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.081351 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d460e938-bd5b-42e4-ad89-f6d8a75e35df-kube-api-access-8c4jd\" (UniqueName: \"kubernetes.io/projected/d460e938-bd5b-42e4-ad89-f6d8a75e35df-kube-api-access-8c4jd\") pod \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " Oct 29 23:11:13.081423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.081407 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-openshift-ca\") pod \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\" (UID: \"d460e938-bd5b-42e4-ad89-f6d8a75e35df\") " Oct 29 23:11:13.082137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.081853 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-openshift-ca" pod "d460e938-bd5b-42e4-ad89-f6d8a75e35df" (UID: "d460e938-bd5b-42e4-ad89-f6d8a75e35df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:13.082272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.082201 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:13.083980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.083956 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d460e938-bd5b-42e4-ad89-f6d8a75e35df-kube-api-access-8c4jd" pod "d460e938-bd5b-42e4-ad89-f6d8a75e35df" (UID: "d460e938-bd5b-42e4-ad89-f6d8a75e35df"). InnerVolumeSpecName "kube-api-access-8c4jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:13.085389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.085357 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-xks-ca" pod "d460e938-bd5b-42e4-ad89-f6d8a75e35df" (UID: "d460e938-bd5b-42e4-ad89-f6d8a75e35df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:13.183387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.183349 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4jd\" (UniqueName: \"kubernetes.io/projected/d460e938-bd5b-42e4-ad89-f6d8a75e35df-kube-api-access-8c4jd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:13.183387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.183382 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460e938-bd5b-42e4-ad89-f6d8a75e35df-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:13.795598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.792640 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" Oct 29 23:11:13.795598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.792648 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr" event={"ID":"d460e938-bd5b-42e4-ad89-f6d8a75e35df","Type":"ContainerDied","Data":"2cfe4e392b1d8810a28270c367b4f4bc44a1de4cea2a15532ea908480132b024"} Oct 29 23:11:13.795598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.792709 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cfe4e392b1d8810a28270c367b4f4bc44a1de4cea2a15532ea908480132b024" Oct 29 23:11:13.958818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:13.958790 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:14.105453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.105370 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0304ee61-d364-4f68-afe6-04af71b0e1df-kube-api-access-zqvvb\" (UniqueName: \"kubernetes.io/projected/0304ee61-d364-4f68-afe6-04af71b0e1df-kube-api-access-zqvvb\") pod \"0304ee61-d364-4f68-afe6-04af71b0e1df\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " Oct 29 23:11:14.105620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.105595 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-xks-ca\") pod \"0304ee61-d364-4f68-afe6-04af71b0e1df\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " Oct 29 23:11:14.105709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.105691 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-openshift-ca\") pod \"0304ee61-d364-4f68-afe6-04af71b0e1df\" (UID: \"0304ee61-d364-4f68-afe6-04af71b0e1df\") " Oct 29 23:11:14.106140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.106105 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-openshift-ca" pod "0304ee61-d364-4f68-afe6-04af71b0e1df" (UID: "0304ee61-d364-4f68-afe6-04af71b0e1df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:14.106358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.106335 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:14.108386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.108365 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0304ee61-d364-4f68-afe6-04af71b0e1df-kube-api-access-zqvvb" pod "0304ee61-d364-4f68-afe6-04af71b0e1df" (UID: "0304ee61-d364-4f68-afe6-04af71b0e1df"). InnerVolumeSpecName "kube-api-access-zqvvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:14.108483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.108425 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-xks-ca" pod "0304ee61-d364-4f68-afe6-04af71b0e1df" (UID: "0304ee61-d364-4f68-afe6-04af71b0e1df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:14.207433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.207403 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqvvb\" (UniqueName: \"kubernetes.io/projected/0304ee61-d364-4f68-afe6-04af71b0e1df-kube-api-access-zqvvb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:14.207433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.207438 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0304ee61-d364-4f68-afe6-04af71b0e1df-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:14.251223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.251191 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7"] Oct 29 23:11:14.255297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.255276 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7af1a5-f74b7"] Oct 29 23:11:14.801535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.801503 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" Oct 29 23:11:14.802035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.801502 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg" event={"ID":"0304ee61-d364-4f68-afe6-04af71b0e1df","Type":"ContainerDied","Data":"acd310e4a0d27fc4e1d2951da7611773ad31e455b25ebc5c8612e7f331ba306e"} Oct 29 23:11:14.802035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:14.801646 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd310e4a0d27fc4e1d2951da7611773ad31e455b25ebc5c8612e7f331ba306e" Oct 29 23:11:15.004416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:15.004379 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3d8c66-cdba-430d-8644-2c7a0a893f94" path="/var/lib/kubelet/pods/5c3d8c66-cdba-430d-8644-2c7a0a893f94/volumes" Oct 29 23:11:19.755602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:19.740510 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx"] Oct 29 23:11:20.038659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.031710 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.129604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.128685 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.158605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.155107 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.158605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.155885 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.172611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.171680 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.286604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.269673 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 23:11:20.407867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.406662 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.686799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.686701 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.713277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.713170 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.713837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.713809 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 23:11:20.714259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.714231 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.788657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.787212 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 29 23:11:20.788657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.787998 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" podUID="db18ea92-9d40-4840-9c15-353d841dea31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.884063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.883497 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.907765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.905687 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 29 23:11:20.907765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.906137 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:20.942605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:20.936157 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.042605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.023350 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.042605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.035414 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.098110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.098077 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:11:21.111762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.111704 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.161605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.155731 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.161605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.156481 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:11:21.161605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.157665 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.161605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.157990 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:11:21.195654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.194613 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.195654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.194993 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 23:11:21.423603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.422339 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.423603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.423040 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 23:11:21.665844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.665797 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.693556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.666284 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.705162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.692418 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.719211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.705739 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:11:21.746213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.719570 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.747609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.720157 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 23:11:21.747756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.747612 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:11:21.747756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.747672 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:11:21.747880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.720186 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 23:11:21.793604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.720693 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.793604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.756642 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:11:21.838603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.834039 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" podUID="db18ea92-9d40-4840-9c15-353d841dea31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.878605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.862688 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 29 23:11:21.878605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.863970 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:11:21.878605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.864031 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 23:11:21.931661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.921447 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.945642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.934784 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:11:21.945642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.935737 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:21.945642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:21.936880 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:11:22.097998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.097101 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.126532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.126505 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:11:22.129607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.129085 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.133435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.132538 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 23:11:22.133435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.132614 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:11:22.198494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.198402 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 23:11:22.199133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.199099 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.199472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.199450 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.256600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.247155 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.278266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.274990 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 23:11:22.289609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.287785 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:11:22.292630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.289913 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:22.412861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.412829 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:11:22.439677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.439646 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 23:11:22.439943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.439928 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:11:22.473658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.465841 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 29 23:11:22.571853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.571823 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 23:11:22.575032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.574884 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 29 23:11:22.586452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.586417 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 23:11:22.588780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.588758 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:11:22.594768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.594172 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:11:22.595284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.595263 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:11:22.596838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.596816 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:11:22.614683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:22.614255 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:11:23.722822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.722788 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz"] Oct 29 23:11:23.723649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.723630 5736 cpu_manager.go:414] "containerMap: removing container" podUID="0304ee61-d364-4f68-afe6-04af71b0e1df" containerName="revoke-keys" Oct 29 23:11:23.723649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.723649 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0304ee61-d364-4f68-afe6-04af71b0e1df" containerName="revoke-keys" Oct 29 23:11:23.723827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.723663 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d460e938-bd5b-42e4-ad89-f6d8a75e35df" containerName="revoke-keys" Oct 29 23:11:23.723827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.723672 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d460e938-bd5b-42e4-ad89-f6d8a75e35df" containerName="revoke-keys" Oct 29 23:11:23.723928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.723847 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="0304ee61-d364-4f68-afe6-04af71b0e1df" containerName="revoke-keys" Oct 29 23:11:23.723928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.723859 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d460e938-bd5b-42e4-ad89-f6d8a75e35df" containerName="revoke-keys" Oct 29 23:11:23.729270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.729250 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:23.744435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.744406 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnjw\" (UniqueName: \"kubernetes.io/projected/ae34499a-24fe-45bc-91db-f2f3efdd9af7-kube-api-access-ncnjw\") pod \"maas-api-revoke-sub-25c86224-x92fz\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:23.744621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.744567 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c86224-x92fz\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:23.744836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.744810 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c86224-x92fz\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:23.746280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.746256 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz"] Oct 29 23:11:23.754351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.754330 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:11:23.754351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.754345 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:11:23.845783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.845748 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c86224-x92fz\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:23.845929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.845904 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnjw\" (UniqueName: \"kubernetes.io/projected/ae34499a-24fe-45bc-91db-f2f3efdd9af7-kube-api-access-ncnjw\") pod \"maas-api-revoke-sub-25c86224-x92fz\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:23.846062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.846041 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c86224-x92fz\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:23.846677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.846653 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c86224-x92fz\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:23.848121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.848100 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c86224-x92fz\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:23.869141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:23.869112 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnjw\" (UniqueName: \"kubernetes.io/projected/ae34499a-24fe-45bc-91db-f2f3efdd9af7-kube-api-access-ncnjw\") pod \"maas-api-revoke-sub-25c86224-x92fz\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:24.043443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.043356 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:24.075549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.075515 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj"] Oct 29 23:11:24.081079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.081052 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.110076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.109659 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj"] Oct 29 23:11:24.150502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.150473 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc91e03f-qxqxj\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.150667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.150532 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xmr\" (UniqueName: \"kubernetes.io/projected/db05b052-2c44-4e6b-950e-a4f1c425e87c-kube-api-access-b2xmr\") pod \"maas-api-revoke-sub-dc91e03f-qxqxj\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.150717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.150694 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc91e03f-qxqxj\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.251889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.251852 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc91e03f-qxqxj\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.252055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.251946 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xmr\" (UniqueName: \"kubernetes.io/projected/db05b052-2c44-4e6b-950e-a4f1c425e87c-kube-api-access-b2xmr\") pod \"maas-api-revoke-sub-dc91e03f-qxqxj\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.252055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.252016 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc91e03f-qxqxj\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.252690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.252668 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc91e03f-qxqxj\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.254266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.254241 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc91e03f-qxqxj\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.275463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.275435 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xmr\" (UniqueName: \"kubernetes.io/projected/db05b052-2c44-4e6b-950e-a4f1c425e87c-kube-api-access-b2xmr\") pod \"maas-api-revoke-sub-dc91e03f-qxqxj\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.353254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.353228 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz"] Oct 29 23:11:24.355015 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:24.354989 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae34499a_24fe_45bc_91db_f2f3efdd9af7.slice/crio-0b0e37d8b38d77336b5591524894832e95b6c5b6bba8eb24746416b32563c774 WatchSource:0}: Error finding container 0b0e37d8b38d77336b5591524894832e95b6c5b6bba8eb24746416b32563c774: Status 404 returned error can't find the container with id 0b0e37d8b38d77336b5591524894832e95b6c5b6bba8eb24746416b32563c774 Oct 29 23:11:24.397443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.397423 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:24.710434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.710400 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj"] Oct 29 23:11:24.713154 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:24.713107 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb05b052_2c44_4e6b_950e_a4f1c425e87c.slice/crio-ed077886e8cfde42e341e44a22ce51106f830205c8211d8929e81e1525fc56b6 WatchSource:0}: Error finding container ed077886e8cfde42e341e44a22ce51106f830205c8211d8929e81e1525fc56b6: Status 404 returned error can't find the container with id ed077886e8cfde42e341e44a22ce51106f830205c8211d8929e81e1525fc56b6 Oct 29 23:11:24.924635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.924551 5736 scope.go:122] "RemoveContainer" containerID="d70da59f1724364147e5f4ab100db6db1e85af5d893e7314b69807aee51e7497" Oct 29 23:11:24.935021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:24.934992 5736 scope.go:122] "RemoveContainer" containerID="eb157e76c8ed89233caff22255392cb3521cd132a3d274170da91dd14a5e7266" Oct 29 23:11:25.164410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:25.164380 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" event={"ID":"ae34499a-24fe-45bc-91db-f2f3efdd9af7","Type":"ContainerStarted","Data":"25f29bebf9fbf5b9e04c6a603c350ef93aefd523547c8377b81eb62a1449a104"} Oct 29 23:11:25.164567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:25.164433 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" event={"ID":"ae34499a-24fe-45bc-91db-f2f3efdd9af7","Type":"ContainerStarted","Data":"0b0e37d8b38d77336b5591524894832e95b6c5b6bba8eb24746416b32563c774"} Oct 29 23:11:25.166237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:25.166207 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" event={"ID":"db05b052-2c44-4e6b-950e-a4f1c425e87c","Type":"ContainerStarted","Data":"2bb0ef51aa902eb315aa2203b1b5cc8772a87d0af00d3afb5edc081c82ede1f7"} Oct 29 23:11:25.166343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:25.166258 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" event={"ID":"db05b052-2c44-4e6b-950e-a4f1c425e87c","Type":"ContainerStarted","Data":"ed077886e8cfde42e341e44a22ce51106f830205c8211d8929e81e1525fc56b6"} Oct 29 23:11:25.205948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:25.205849 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" podStartSLOduration=2.205833386 podStartE2EDuration="2.205833386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:25.203743328 +0000 UTC m=+1682.319662939" watchObservedRunningTime="2026-10-29 23:11:25.205833386 +0000 UTC m=+1682.321752991" Oct 29 23:11:25.246084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:25.246025 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" podStartSLOduration=1.2460069599999999 podStartE2EDuration="1.24600696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:25.243948118 +0000 UTC m=+1682.359867725" watchObservedRunningTime="2026-10-29 23:11:25.24600696 +0000 UTC m=+1682.361926567" Oct 29 23:11:26.175910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:26.175875 5736 generic.go:356] "Generic (PLEG): container finished" podID="db05b052-2c44-4e6b-950e-a4f1c425e87c" containerID="2bb0ef51aa902eb315aa2203b1b5cc8772a87d0af00d3afb5edc081c82ede1f7" exitCode=0 Oct 29 23:11:26.176352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:26.175953 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" event={"ID":"db05b052-2c44-4e6b-950e-a4f1c425e87c","Type":"ContainerDied","Data":"2bb0ef51aa902eb315aa2203b1b5cc8772a87d0af00d3afb5edc081c82ede1f7"} Oct 29 23:11:26.177826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:26.177800 5736 generic.go:356] "Generic (PLEG): container finished" podID="ae34499a-24fe-45bc-91db-f2f3efdd9af7" containerID="25f29bebf9fbf5b9e04c6a603c350ef93aefd523547c8377b81eb62a1449a104" exitCode=0 Oct 29 23:11:26.177961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:26.177888 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" event={"ID":"ae34499a-24fe-45bc-91db-f2f3efdd9af7","Type":"ContainerDied","Data":"25f29bebf9fbf5b9e04c6a603c350ef93aefd523547c8377b81eb62a1449a104"} Oct 29 23:11:26.824096 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:11:26.824057 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:11:27.364627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.364566 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:27.365004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.364784 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf"] Oct 29 23:11:27.365068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.365009 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" containerID="cri-o://60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e" gracePeriod=30 Oct 29 23:11:27.397821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.397797 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:27.398170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.398148 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae34499a-24fe-45bc-91db-f2f3efdd9af7-kube-api-access-ncnjw\" (UniqueName: \"kubernetes.io/projected/ae34499a-24fe-45bc-91db-f2f3efdd9af7-kube-api-access-ncnjw\") pod \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " Oct 29 23:11:27.398259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.398244 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-xks-ca\") pod \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " Oct 29 23:11:27.398458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.398411 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-openshift-ca\") pod \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\" (UID: \"ae34499a-24fe-45bc-91db-f2f3efdd9af7\") " Oct 29 23:11:27.398789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.398756 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-openshift-ca" pod "ae34499a-24fe-45bc-91db-f2f3efdd9af7" (UID: "ae34499a-24fe-45bc-91db-f2f3efdd9af7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:27.399428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.399407 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:27.400429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.400401 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae34499a-24fe-45bc-91db-f2f3efdd9af7-kube-api-access-ncnjw" pod "ae34499a-24fe-45bc-91db-f2f3efdd9af7" (UID: "ae34499a-24fe-45bc-91db-f2f3efdd9af7"). InnerVolumeSpecName "kube-api-access-ncnjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:27.400752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.400728 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-xks-ca" pod "ae34499a-24fe-45bc-91db-f2f3efdd9af7" (UID: "ae34499a-24fe-45bc-91db-f2f3efdd9af7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:27.500709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.500672 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-xks-ca\") pod \"db05b052-2c44-4e6b-950e-a4f1c425e87c\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " Oct 29 23:11:27.500884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.500796 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-openshift-ca\") pod \"db05b052-2c44-4e6b-950e-a4f1c425e87c\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " Oct 29 23:11:27.501038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.501017 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db05b052-2c44-4e6b-950e-a4f1c425e87c-kube-api-access-b2xmr\" (UniqueName: \"kubernetes.io/projected/db05b052-2c44-4e6b-950e-a4f1c425e87c-kube-api-access-b2xmr\") pod \"db05b052-2c44-4e6b-950e-a4f1c425e87c\" (UID: \"db05b052-2c44-4e6b-950e-a4f1c425e87c\") " Oct 29 23:11:27.501223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.501197 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-openshift-ca" pod "db05b052-2c44-4e6b-950e-a4f1c425e87c" (UID: "db05b052-2c44-4e6b-950e-a4f1c425e87c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:27.501548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.501529 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae34499a-24fe-45bc-91db-f2f3efdd9af7-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:27.501629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.501554 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:27.501629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.501569 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnjw\" (UniqueName: \"kubernetes.io/projected/ae34499a-24fe-45bc-91db-f2f3efdd9af7-kube-api-access-ncnjw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:27.502830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.502811 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-xks-ca" pod "db05b052-2c44-4e6b-950e-a4f1c425e87c" (UID: "db05b052-2c44-4e6b-950e-a4f1c425e87c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:27.503049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.503027 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db05b052-2c44-4e6b-950e-a4f1c425e87c-kube-api-access-b2xmr" pod "db05b052-2c44-4e6b-950e-a4f1c425e87c" (UID: "db05b052-2c44-4e6b-950e-a4f1c425e87c"). InnerVolumeSpecName "kube-api-access-b2xmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:27.603407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.603377 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db05b052-2c44-4e6b-950e-a4f1c425e87c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:27.603407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:27.603407 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xmr\" (UniqueName: \"kubernetes.io/projected/db05b052-2c44-4e6b-950e-a4f1c425e87c-kube-api-access-b2xmr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:28.194960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:28.194930 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" Oct 29 23:11:28.195136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:28.194969 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz" event={"ID":"ae34499a-24fe-45bc-91db-f2f3efdd9af7","Type":"ContainerDied","Data":"0b0e37d8b38d77336b5591524894832e95b6c5b6bba8eb24746416b32563c774"} Oct 29 23:11:28.195136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:28.195017 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b0e37d8b38d77336b5591524894832e95b6c5b6bba8eb24746416b32563c774" Oct 29 23:11:28.196485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:28.196458 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" event={"ID":"db05b052-2c44-4e6b-950e-a4f1c425e87c","Type":"ContainerDied","Data":"ed077886e8cfde42e341e44a22ce51106f830205c8211d8929e81e1525fc56b6"} Oct 29 23:11:28.196622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:28.196504 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj" Oct 29 23:11:28.196622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:28.196505 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed077886e8cfde42e341e44a22ce51106f830205c8211d8929e81e1525fc56b6" Oct 29 23:11:31.044725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:31.044690 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp"] Oct 29 23:11:31.045395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:31.044869 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" containerID="cri-o://139b627c7e0cea3285bafc0b9fd5c694ffe48710ea2e717f6c66540ccf16d15a" gracePeriod=30 Oct 29 23:11:32.676305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.676280 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 23:11:32.792610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.792563 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-socket\") pod \"870c69cc-31f5-45a3-84a3-113138e2c8ee\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " Oct 29 23:11:32.792767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.792652 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-token\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-token\") pod \"870c69cc-31f5-45a3-84a3-113138e2c8ee\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " Oct 29 23:11:32.792767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.792697 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-kube-api-access-tz5kg\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-kube-api-access-tz5kg\") pod \"870c69cc-31f5-45a3-84a3-113138e2c8ee\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " Oct 29 23:11:32.792848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.792825 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870c69cc-31f5-45a3-84a3-113138e2c8ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870c69cc-31f5-45a3-84a3-113138e2c8ee-istiod-ca-cert\") pod \"870c69cc-31f5-45a3-84a3-113138e2c8ee\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " Oct 29 23:11:32.792848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.792830 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-socket" pod "870c69cc-31f5-45a3-84a3-113138e2c8ee" (UID: "870c69cc-31f5-45a3-84a3-113138e2c8ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:32.792939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.792907 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-certs\") pod \"870c69cc-31f5-45a3-84a3-113138e2c8ee\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " Oct 29 23:11:32.793004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.792960 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-envoy\") pod \"870c69cc-31f5-45a3-84a3-113138e2c8ee\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " Oct 29 23:11:32.793004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.792989 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-data\") pod \"870c69cc-31f5-45a3-84a3-113138e2c8ee\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " Oct 29 23:11:32.793113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793059 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-credential-socket\") pod \"870c69cc-31f5-45a3-84a3-113138e2c8ee\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " Oct 29 23:11:32.793189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793113 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-podinfo\") pod \"870c69cc-31f5-45a3-84a3-113138e2c8ee\" (UID: \"870c69cc-31f5-45a3-84a3-113138e2c8ee\") " Oct 29 23:11:32.793189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793114 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-certs" pod "870c69cc-31f5-45a3-84a3-113138e2c8ee" (UID: "870c69cc-31f5-45a3-84a3-113138e2c8ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:32.793346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793232 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870c69cc-31f5-45a3-84a3-113138e2c8ee-istiod-ca-cert" pod "870c69cc-31f5-45a3-84a3-113138e2c8ee" (UID: "870c69cc-31f5-45a3-84a3-113138e2c8ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:32.793460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793413 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-data" pod "870c69cc-31f5-45a3-84a3-113138e2c8ee" (UID: "870c69cc-31f5-45a3-84a3-113138e2c8ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:32.793532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793494 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-credential-socket" pod "870c69cc-31f5-45a3-84a3-113138e2c8ee" (UID: "870c69cc-31f5-45a3-84a3-113138e2c8ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:32.793940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793918 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870c69cc-31f5-45a3-84a3-113138e2c8ee-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:32.794025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793948 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:32.794025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793962 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:32.794025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793975 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:32.794025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.793991 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:32.795080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.795033 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-token" pod "870c69cc-31f5-45a3-84a3-113138e2c8ee" (UID: "870c69cc-31f5-45a3-84a3-113138e2c8ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:32.795198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.795127 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-envoy" pod "870c69cc-31f5-45a3-84a3-113138e2c8ee" (UID: "870c69cc-31f5-45a3-84a3-113138e2c8ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:32.795406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.795363 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-podinfo" pod "870c69cc-31f5-45a3-84a3-113138e2c8ee" (UID: "870c69cc-31f5-45a3-84a3-113138e2c8ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:32.795406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.795366 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-kube-api-access-tz5kg" pod "870c69cc-31f5-45a3-84a3-113138e2c8ee" (UID: "870c69cc-31f5-45a3-84a3-113138e2c8ee"). InnerVolumeSpecName "kube-api-access-tz5kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:32.895750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.895718 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:32.895750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.895746 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:32.895750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.895755 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz5kg\" (UniqueName: \"kubernetes.io/projected/870c69cc-31f5-45a3-84a3-113138e2c8ee-kube-api-access-tz5kg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:32.896007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:32.895764 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870c69cc-31f5-45a3-84a3-113138e2c8ee-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:33.241788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:33.241744 5736 generic.go:356] "Generic (PLEG): container finished" podID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerID="60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e" exitCode=0 Oct 29 23:11:33.241976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:33.241818 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" event={"ID":"870c69cc-31f5-45a3-84a3-113138e2c8ee","Type":"ContainerDied","Data":"60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e"} Oct 29 23:11:33.241976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:33.241901 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" event={"ID":"870c69cc-31f5-45a3-84a3-113138e2c8ee","Type":"ContainerDied","Data":"ed11f61de045780c103a1cca184886f92147fe4c9458dd822d7ae93ada945d87"} Oct 29 23:11:33.241976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:33.241930 5736 scope.go:122] "RemoveContainer" containerID="60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e" Oct 29 23:11:33.242134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:33.242118 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf" Oct 29 23:11:33.259929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:33.259899 5736 scope.go:122] "RemoveContainer" containerID="60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e" Oct 29 23:11:33.260473 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:11:33.260447 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e\": container with ID starting with 60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e not found: ID does not exist" containerID="60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e" Oct 29 23:11:33.260630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:33.260479 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e"} err="failed to get container status \"60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e\": rpc error: code = NotFound desc = could not find container \"60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e\": container with ID starting with 60b74e54b4bd821d0c9c18737cc563936cf06504a878126fd6be13a5922f595e not found: ID does not exist" Oct 29 23:11:33.299911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:33.299838 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf"] Oct 29 23:11:33.307849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:33.307801 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-qj6tf"] Oct 29 23:11:35.005497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:35.005454 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" path="/var/lib/kubelet/pods/870c69cc-31f5-45a3-84a3-113138e2c8ee/volumes" Oct 29 23:11:35.006028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:35.005806 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m"] Oct 29 23:11:35.017937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:35.017900 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-31d798-88n9m"] Oct 29 23:11:36.268674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.268629 5736 generic.go:356] "Generic (PLEG): container finished" podID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerID="139b627c7e0cea3285bafc0b9fd5c694ffe48710ea2e717f6c66540ccf16d15a" exitCode=0 Oct 29 23:11:36.269159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.268700 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" event={"ID":"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4","Type":"ContainerDied","Data":"139b627c7e0cea3285bafc0b9fd5c694ffe48710ea2e717f6c66540ccf16d15a"} Oct 29 23:11:36.315005 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.314939 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 23:11:36.468600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.462993 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-socket\") pod \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " Oct 29 23:11:36.468600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.463059 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istiod-ca-cert\") pod \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " Oct 29 23:11:36.468600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.463098 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-kube-api-access-vttgp\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-kube-api-access-vttgp\") pod \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " Oct 29 23:11:36.468600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.463123 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-podinfo\") pod \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " Oct 29 23:11:36.468600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.463243 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-certs\") pod \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " Oct 29 23:11:36.468600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.463285 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-data\") pod \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " Oct 29 23:11:36.468600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.463421 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-token\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-token\") pod \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " Oct 29 23:11:36.468600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.463517 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-credential-socket\") pod \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " Oct 29 23:11:36.480463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.480410 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-data" pod "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" (UID: "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:36.480936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.480912 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-certs" pod "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" (UID: "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:36.481907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.481884 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-envoy\") pod \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\" (UID: \"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4\") " Oct 29 23:11:36.482800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.482777 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:36.483370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.483354 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:36.483836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.483623 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istiod-ca-cert" pod "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" (UID: "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:36.484230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.484208 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-socket" pod "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" (UID: "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:36.484565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.484547 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-credential-socket" pod "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" (UID: "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:36.523570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.523066 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-token" pod "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" (UID: "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:36.523570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.523225 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-podinfo" pod "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" (UID: "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:36.524051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.523981 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-envoy" pod "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" (UID: "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:36.527611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.525031 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-kube-api-access-vttgp" pod "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" (UID: "5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4"). InnerVolumeSpecName "kube-api-access-vttgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:36.585319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.585249 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:36.585319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.585278 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:36.585319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.585288 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:36.585319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.585298 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vttgp\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-kube-api-access-vttgp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:36.585319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.585308 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:36.585319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.585318 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:36.585654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:36.585326 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.003591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.003548 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d679de3-e7cd-4fca-9951-3a26ace94acf" path="/var/lib/kubelet/pods/3d679de3-e7cd-4fca-9951-3a26ace94acf/volumes" Oct 29 23:11:37.232521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.232466 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" podUID="db18ea92-9d40-4840-9c15-353d841dea31" containerName="main" containerID="cri-o://8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace" gracePeriod=43 Oct 29 23:11:37.277287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.277215 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" event={"ID":"5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4","Type":"ContainerDied","Data":"b12331ee990f9bbaa9d8111ad87763e90c0fced81d043267ee95897e7ea539b4"} Oct 29 23:11:37.277287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.277285 5736 scope.go:122] "RemoveContainer" containerID="139b627c7e0cea3285bafc0b9fd5c694ffe48710ea2e717f6c66540ccf16d15a" Oct 29 23:11:37.277669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.277445 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp" Oct 29 23:11:37.348871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.348850 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp"] Oct 29 23:11:37.360184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.360159 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-hcnbp"] Oct 29 23:11:37.462904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.462858 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7"] Oct 29 23:11:37.463762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.463737 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ae34499a-24fe-45bc-91db-f2f3efdd9af7" containerName="revoke-keys" Oct 29 23:11:37.463762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.463763 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae34499a-24fe-45bc-91db-f2f3efdd9af7" containerName="revoke-keys" Oct 29 23:11:37.463949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.463802 5736 cpu_manager.go:414] "containerMap: removing container" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" Oct 29 23:11:37.463949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.463811 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" Oct 29 23:11:37.463949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.463843 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" Oct 29 23:11:37.463949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.463850 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" Oct 29 23:11:37.463949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.463863 5736 cpu_manager.go:414] "containerMap: removing container" podUID="db05b052-2c44-4e6b-950e-a4f1c425e87c" containerName="revoke-keys" Oct 29 23:11:37.463949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.463870 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db05b052-2c44-4e6b-950e-a4f1c425e87c" containerName="revoke-keys" Oct 29 23:11:37.464264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.464026 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" containerName="istio-proxy" Oct 29 23:11:37.464264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.464043 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="db05b052-2c44-4e6b-950e-a4f1c425e87c" containerName="revoke-keys" Oct 29 23:11:37.464264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.464059 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae34499a-24fe-45bc-91db-f2f3efdd9af7" containerName="revoke-keys" Oct 29 23:11:37.464264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.464074 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="870c69cc-31f5-45a3-84a3-113138e2c8ee" containerName="istio-proxy" Oct 29 23:11:37.470964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.470939 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.484837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.484816 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7"] Oct 29 23:11:37.486224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.486197 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:11:37.486336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.486264 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:11:37.496811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.496782 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2mwm\" (UniqueName: \"kubernetes.io/projected/e1e539e5-7b25-449f-bfcd-122b9b3642ab-kube-api-access-r2mwm\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.496925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.496851 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.496996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.496969 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.516607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.516560 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.597973 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-dshm\") pod \"db18ea92-9d40-4840-9c15-353d841dea31\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.598224 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db18ea92-9d40-4840-9c15-353d841dea31-kube-api-access-b6dfr\" (UniqueName: \"kubernetes.io/projected/db18ea92-9d40-4840-9c15-353d841dea31-kube-api-access-b6dfr\") pod \"db18ea92-9d40-4840-9c15-353d841dea31\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.598291 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-tmp-dir\") pod \"db18ea92-9d40-4840-9c15-353d841dea31\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.598432 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-home\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-home\") pod \"db18ea92-9d40-4840-9c15-353d841dea31\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.598501 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-model-cache\") pod \"db18ea92-9d40-4840-9c15-353d841dea31\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.598631 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db18ea92-9d40-4840-9c15-353d841dea31-tls-certs\" (UniqueName: \"kubernetes.io/secret/db18ea92-9d40-4840-9c15-353d841dea31-tls-certs\") pod \"db18ea92-9d40-4840-9c15-353d841dea31\" (UID: \"db18ea92-9d40-4840-9c15-353d841dea31\") " Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.599432 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2mwm\" (UniqueName: \"kubernetes.io/projected/e1e539e5-7b25-449f-bfcd-122b9b3642ab-kube-api-access-r2mwm\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.599504 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.599540 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.600262 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-home" pod "db18ea92-9d40-4840-9c15-353d841dea31" (UID: "db18ea92-9d40-4840-9c15-353d841dea31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.601657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.600941 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.602413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.601684 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-model-cache" pod "db18ea92-9d40-4840-9c15-353d841dea31" (UID: "db18ea92-9d40-4840-9c15-353d841dea31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.605754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.602676 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-tmp-dir" pod "db18ea92-9d40-4840-9c15-353d841dea31" (UID: "db18ea92-9d40-4840-9c15-353d841dea31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.606004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.605977 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db18ea92-9d40-4840-9c15-353d841dea31-tls-certs" pod "db18ea92-9d40-4840-9c15-353d841dea31" (UID: "db18ea92-9d40-4840-9c15-353d841dea31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:37.610596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.609177 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.612688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.612659 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-dshm" pod "db18ea92-9d40-4840-9c15-353d841dea31" (UID: "db18ea92-9d40-4840-9c15-353d841dea31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.613018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.612990 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db18ea92-9d40-4840-9c15-353d841dea31-kube-api-access-b6dfr" pod "db18ea92-9d40-4840-9c15-353d841dea31" (UID: "db18ea92-9d40-4840-9c15-353d841dea31"). InnerVolumeSpecName "kube-api-access-b6dfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:37.629206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.629184 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2mwm\" (UniqueName: \"kubernetes.io/projected/e1e539e5-7b25-449f-bfcd-122b9b3642ab-kube-api-access-r2mwm\") pod \"maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:37.701564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.701526 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.701564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.701558 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.701564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.701569 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db18ea92-9d40-4840-9c15-353d841dea31-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.701819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.701612 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.701819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.701622 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6dfr\" (UniqueName: \"kubernetes.io/projected/db18ea92-9d40-4840-9c15-353d841dea31-kube-api-access-b6dfr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.701819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.701634 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db18ea92-9d40-4840-9c15-353d841dea31-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.814865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:37.814833 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:38.118665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.118629 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h"] Oct 29 23:11:38.119620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.119558 5736 cpu_manager.go:414] "containerMap: removing container" podUID="db18ea92-9d40-4840-9c15-353d841dea31" containerName="main" Oct 29 23:11:38.119886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.119808 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db18ea92-9d40-4840-9c15-353d841dea31" containerName="main" Oct 29 23:11:38.120083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.120066 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="db18ea92-9d40-4840-9c15-353d841dea31" containerName="main" Oct 29 23:11:38.125059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.125039 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7"] Oct 29 23:11:38.125158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.125132 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.126171 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:38.126147 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e539e5_7b25_449f_bfcd_122b9b3642ab.slice/crio-146506cbc5d65e08052626d0cf765160dc9fd63f071ff0f1fe5dceafbf4d01cc WatchSource:0}: Error finding container 146506cbc5d65e08052626d0cf765160dc9fd63f071ff0f1fe5dceafbf4d01cc: Status 404 returned error can't find the container with id 146506cbc5d65e08052626d0cf765160dc9fd63f071ff0f1fe5dceafbf4d01cc Oct 29 23:11:38.144430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.144304 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h"] Oct 29 23:11:38.208840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.208811 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52b310a3-tvg2h\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.208970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.208848 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snqtq\" (UniqueName: \"kubernetes.io/projected/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-kube-api-access-snqtq\") pod \"maas-api-revoke-sub-52b310a3-tvg2h\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.208970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.208956 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52b310a3-tvg2h\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.286555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.286523 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" event={"ID":"e1e539e5-7b25-449f-bfcd-122b9b3642ab","Type":"ContainerStarted","Data":"146506cbc5d65e08052626d0cf765160dc9fd63f071ff0f1fe5dceafbf4d01cc"} Oct 29 23:11:38.289250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.289226 5736 generic.go:356] "Generic (PLEG): container finished" podID="db18ea92-9d40-4840-9c15-353d841dea31" containerID="8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace" exitCode=0 Oct 29 23:11:38.289389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.289278 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" Oct 29 23:11:38.289389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.289300 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" event={"ID":"db18ea92-9d40-4840-9c15-353d841dea31","Type":"ContainerDied","Data":"8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace"} Oct 29 23:11:38.289389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.289335 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx" event={"ID":"db18ea92-9d40-4840-9c15-353d841dea31","Type":"ContainerDied","Data":"a15f4a549ef3582041574a336126f0f9fd98228acdb61e504ea96f07072ff28a"} Oct 29 23:11:38.289389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.289359 5736 scope.go:122] "RemoveContainer" containerID="8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace" Oct 29 23:11:38.301696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.301677 5736 scope.go:122] "RemoveContainer" containerID="8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace" Oct 29 23:11:38.301951 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:11:38.301932 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace\": container with ID starting with 8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace not found: ID does not exist" containerID="8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace" Oct 29 23:11:38.301992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.301961 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace"} err="failed to get container status \"8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace\": rpc error: code = NotFound desc = could not find container \"8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace\": container with ID starting with 8842568a6fe8ab09e18f6fba6473521605c7a3e1b7bd1e9ae1cbb276d9c0cace not found: ID does not exist" Oct 29 23:11:38.310767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.310744 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52b310a3-tvg2h\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.310881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.310775 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snqtq\" (UniqueName: \"kubernetes.io/projected/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-kube-api-access-snqtq\") pod \"maas-api-revoke-sub-52b310a3-tvg2h\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.310881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.310830 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52b310a3-tvg2h\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.311411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.311395 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52b310a3-tvg2h\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.312871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.312852 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52b310a3-tvg2h\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.334378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.334357 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snqtq\" (UniqueName: \"kubernetes.io/projected/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-kube-api-access-snqtq\") pod \"maas-api-revoke-sub-52b310a3-tvg2h\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.351317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.351291 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx"] Oct 29 23:11:38.354629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.354561 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7af1a5/ipp-route-7af1a5-kserve-5555f84b85-fcdpx"] Oct 29 23:11:38.482960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.482927 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:38.790357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:38.790328 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h"] Oct 29 23:11:38.792753 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:38.792722 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf11f4_bac4_4a7e_8d7d_0b3f49c31d0f.slice/crio-bc0c34bd80922e1b1346f5148c89c0b0540581d2339b37df7435a733cc2c2408 WatchSource:0}: Error finding container bc0c34bd80922e1b1346f5148c89c0b0540581d2339b37df7435a733cc2c2408: Status 404 returned error can't find the container with id bc0c34bd80922e1b1346f5148c89c0b0540581d2339b37df7435a733cc2c2408 Oct 29 23:11:39.004719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:39.004689 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4" path="/var/lib/kubelet/pods/5c5432e2-4ef8-41ce-9f7c-8e01ac9bcca4/volumes" Oct 29 23:11:39.005204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:39.005184 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db18ea92-9d40-4840-9c15-353d841dea31" path="/var/lib/kubelet/pods/db18ea92-9d40-4840-9c15-353d841dea31/volumes" Oct 29 23:11:39.301521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:39.301440 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" event={"ID":"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f","Type":"ContainerStarted","Data":"174d20062a9018b7ec1dcd62e13f8c9aacdfa080379ec56c7fd68ddcd93e3dfa"} Oct 29 23:11:39.301521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:39.301501 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" event={"ID":"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f","Type":"ContainerStarted","Data":"bc0c34bd80922e1b1346f5148c89c0b0540581d2339b37df7435a733cc2c2408"} Oct 29 23:11:39.302970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:39.302946 5736 generic.go:356] "Generic (PLEG): container finished" podID="e1e539e5-7b25-449f-bfcd-122b9b3642ab" containerID="7f90132aacacb74a1812587e201efc76ff0b5257e99d43c6afe4a6e4352e6d39" exitCode=0 Oct 29 23:11:39.303093 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:39.302993 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" event={"ID":"e1e539e5-7b25-449f-bfcd-122b9b3642ab","Type":"ContainerDied","Data":"7f90132aacacb74a1812587e201efc76ff0b5257e99d43c6afe4a6e4352e6d39"} Oct 29 23:11:39.342525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:39.342474 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" podStartSLOduration=1.342460324 podStartE2EDuration="1.342460324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:39.341788019 +0000 UTC m=+1696.457707636" watchObservedRunningTime="2026-10-29 23:11:39.342460324 +0000 UTC m=+1696.458379929" Oct 29 23:11:40.311730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.311690 5736 generic.go:356] "Generic (PLEG): container finished" podID="badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f" containerID="174d20062a9018b7ec1dcd62e13f8c9aacdfa080379ec56c7fd68ddcd93e3dfa" exitCode=0 Oct 29 23:11:40.312148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.311776 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" event={"ID":"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f","Type":"ContainerDied","Data":"174d20062a9018b7ec1dcd62e13f8c9aacdfa080379ec56c7fd68ddcd93e3dfa"} Oct 29 23:11:40.489072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.489046 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:40.541137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.541099 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-xks-ca\") pod \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " Oct 29 23:11:40.541326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.541266 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e539e5-7b25-449f-bfcd-122b9b3642ab-kube-api-access-r2mwm\" (UniqueName: \"kubernetes.io/projected/e1e539e5-7b25-449f-bfcd-122b9b3642ab-kube-api-access-r2mwm\") pod \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " Oct 29 23:11:40.541411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.541334 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-openshift-ca\") pod \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\" (UID: \"e1e539e5-7b25-449f-bfcd-122b9b3642ab\") " Oct 29 23:11:40.541843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.541812 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-openshift-ca" pod "e1e539e5-7b25-449f-bfcd-122b9b3642ab" (UID: "e1e539e5-7b25-449f-bfcd-122b9b3642ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:40.542056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.542012 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:40.543764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.543736 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-xks-ca" pod "e1e539e5-7b25-449f-bfcd-122b9b3642ab" (UID: "e1e539e5-7b25-449f-bfcd-122b9b3642ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:40.543867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.543740 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e539e5-7b25-449f-bfcd-122b9b3642ab-kube-api-access-r2mwm" pod "e1e539e5-7b25-449f-bfcd-122b9b3642ab" (UID: "e1e539e5-7b25-449f-bfcd-122b9b3642ab"). InnerVolumeSpecName "kube-api-access-r2mwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:40.643404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.643332 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e539e5-7b25-449f-bfcd-122b9b3642ab-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:40.643404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:40.643359 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2mwm\" (UniqueName: \"kubernetes.io/projected/e1e539e5-7b25-449f-bfcd-122b9b3642ab-kube-api-access-r2mwm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:41.324781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.324697 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" Oct 29 23:11:41.324781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.324734 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7" event={"ID":"e1e539e5-7b25-449f-bfcd-122b9b3642ab","Type":"ContainerDied","Data":"146506cbc5d65e08052626d0cf765160dc9fd63f071ff0f1fe5dceafbf4d01cc"} Oct 29 23:11:41.325261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.324791 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="146506cbc5d65e08052626d0cf765160dc9fd63f071ff0f1fe5dceafbf4d01cc" Oct 29 23:11:41.490314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.490289 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:41.558896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.558866 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-kube-api-access-snqtq\" (UniqueName: \"kubernetes.io/projected/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-kube-api-access-snqtq\") pod \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " Oct 29 23:11:41.559083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.558923 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-xks-ca\") pod \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " Oct 29 23:11:41.559083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.559041 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-openshift-ca\") pod \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\" (UID: \"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f\") " Oct 29 23:11:41.559830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.559524 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-openshift-ca" pod "badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f" (UID: "badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:41.559830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.559799 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:41.561662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.561636 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-xks-ca" pod "badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f" (UID: "badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:41.561732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.561675 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-kube-api-access-snqtq" pod "badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f" (UID: "badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f"). InnerVolumeSpecName "kube-api-access-snqtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:41.660958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.660877 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snqtq\" (UniqueName: \"kubernetes.io/projected/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-kube-api-access-snqtq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:41.660958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.660907 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:41.988350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.988313 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj"] Oct 29 23:11:41.988925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.988909 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e1e539e5-7b25-449f-bfcd-122b9b3642ab" containerName="revoke-keys" Oct 29 23:11:41.988991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.988927 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e539e5-7b25-449f-bfcd-122b9b3642ab" containerName="revoke-keys" Oct 29 23:11:41.988991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.988968 5736 cpu_manager.go:414] "containerMap: removing container" podUID="badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f" containerName="revoke-keys" Oct 29 23:11:41.988991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.988974 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f" containerName="revoke-keys" Oct 29 23:11:41.989096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.989062 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e539e5-7b25-449f-bfcd-122b9b3642ab" containerName="revoke-keys" Oct 29 23:11:41.989096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.989080 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f" containerName="revoke-keys" Oct 29 23:11:41.992627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:41.992609 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.013166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.013134 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj"] Oct 29 23:11:42.066497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.066459 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjdx\" (UniqueName: \"kubernetes.io/projected/2fdcf95e-b463-494d-8ebc-5b04b812fb19-kube-api-access-zrjdx\") pod \"maas-api-revoke-sub-f5b933b9-r5rpj\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.066693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.066655 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5b933b9-r5rpj\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.066770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.066757 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5b933b9-r5rpj\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.167856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.167826 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjdx\" (UniqueName: \"kubernetes.io/projected/2fdcf95e-b463-494d-8ebc-5b04b812fb19-kube-api-access-zrjdx\") pod \"maas-api-revoke-sub-f5b933b9-r5rpj\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.168041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.167927 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5b933b9-r5rpj\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.168041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.168017 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5b933b9-r5rpj\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.168780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.168755 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5b933b9-r5rpj\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.170280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.170259 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5b933b9-r5rpj\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.192126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.192095 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjdx\" (UniqueName: \"kubernetes.io/projected/2fdcf95e-b463-494d-8ebc-5b04b812fb19-kube-api-access-zrjdx\") pod \"maas-api-revoke-sub-f5b933b9-r5rpj\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.308294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.308213 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:42.339879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.339787 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" Oct 29 23:11:42.339879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.339796 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h" event={"ID":"badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f","Type":"ContainerDied","Data":"bc0c34bd80922e1b1346f5148c89c0b0540581d2339b37df7435a733cc2c2408"} Oct 29 23:11:42.339879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.339853 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc0c34bd80922e1b1346f5148c89c0b0540581d2339b37df7435a733cc2c2408" Oct 29 23:11:42.630658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.624625 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj"] Oct 29 23:11:42.670619 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:42.670508 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdcf95e_b463_494d_8ebc_5b04b812fb19.slice/crio-6f487dfd671360a3c0cf9bd6ce91126f3846f7754dd8a5e76186acc129c34287 WatchSource:0}: Error finding container 6f487dfd671360a3c0cf9bd6ce91126f3846f7754dd8a5e76186acc129c34287: Status 404 returned error can't find the container with id 6f487dfd671360a3c0cf9bd6ce91126f3846f7754dd8a5e76186acc129c34287 Oct 29 23:11:42.976416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.976383 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5"] Oct 29 23:11:42.981719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:42.981696 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.016547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.016510 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5"] Oct 29 23:11:43.093447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.093418 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05eff51a-h5bw5\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.093641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.093467 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05eff51a-h5bw5\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.093734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.093654 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fks\" (UniqueName: \"kubernetes.io/projected/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-kube-api-access-d7fks\") pod \"maas-api-revoke-sub-05eff51a-h5bw5\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.194786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.194745 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05eff51a-h5bw5\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.195023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.194878 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fks\" (UniqueName: \"kubernetes.io/projected/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-kube-api-access-d7fks\") pod \"maas-api-revoke-sub-05eff51a-h5bw5\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.195023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.195010 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05eff51a-h5bw5\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.195489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.195459 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05eff51a-h5bw5\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.197287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.197265 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05eff51a-h5bw5\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.218216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.218189 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fks\" (UniqueName: \"kubernetes.io/projected/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-kube-api-access-d7fks\") pod \"maas-api-revoke-sub-05eff51a-h5bw5\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.295092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.295063 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:43.351996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.351958 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" event={"ID":"2fdcf95e-b463-494d-8ebc-5b04b812fb19","Type":"ContainerStarted","Data":"295495f47a2ff8e62732b2e84a56a9061111e55a007f27d9f33d2744e3d92382"} Oct 29 23:11:43.352418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.352024 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" event={"ID":"2fdcf95e-b463-494d-8ebc-5b04b812fb19","Type":"ContainerStarted","Data":"6f487dfd671360a3c0cf9bd6ce91126f3846f7754dd8a5e76186acc129c34287"} Oct 29 23:11:43.394548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.394496 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" podStartSLOduration=2.3944779179999998 podStartE2EDuration="2.394477918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:43.392092688 +0000 UTC m=+1700.508012294" watchObservedRunningTime="2026-10-29 23:11:43.394477918 +0000 UTC m=+1700.510397524" Oct 29 23:11:43.609964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:43.609938 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5"] Oct 29 23:11:43.612771 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:43.612737 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53b6ac4_e35b_43fa_91cc_f95db9196fc9.slice/crio-dfb8fba5f08de3f20b2321bbfcfe8d621cd23b42a9a154d8334840be12198ca1 WatchSource:0}: Error finding container dfb8fba5f08de3f20b2321bbfcfe8d621cd23b42a9a154d8334840be12198ca1: Status 404 returned error can't find the container with id dfb8fba5f08de3f20b2321bbfcfe8d621cd23b42a9a154d8334840be12198ca1 Oct 29 23:11:44.361450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:44.361412 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" event={"ID":"e53b6ac4-e35b-43fa-91cc-f95db9196fc9","Type":"ContainerStarted","Data":"5173f5474ca753af0995474f2566a65993b9baf42047cfd39c59596fcc8b1a4e"} Oct 29 23:11:44.361982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:44.361479 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" event={"ID":"e53b6ac4-e35b-43fa-91cc-f95db9196fc9","Type":"ContainerStarted","Data":"dfb8fba5f08de3f20b2321bbfcfe8d621cd23b42a9a154d8334840be12198ca1"} Oct 29 23:11:44.363301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:44.363275 5736 generic.go:356] "Generic (PLEG): container finished" podID="2fdcf95e-b463-494d-8ebc-5b04b812fb19" containerID="295495f47a2ff8e62732b2e84a56a9061111e55a007f27d9f33d2744e3d92382" exitCode=0 Oct 29 23:11:44.363394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:44.363317 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" event={"ID":"2fdcf95e-b463-494d-8ebc-5b04b812fb19","Type":"ContainerDied","Data":"295495f47a2ff8e62732b2e84a56a9061111e55a007f27d9f33d2744e3d92382"} Oct 29 23:11:44.403757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:44.403707 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" podStartSLOduration=2.403691053 podStartE2EDuration="2.403691053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:44.401272527 +0000 UTC m=+1701.517192133" watchObservedRunningTime="2026-10-29 23:11:44.403691053 +0000 UTC m=+1701.519610660" Oct 29 23:11:45.373908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.373872 5736 generic.go:356] "Generic (PLEG): container finished" podID="e53b6ac4-e35b-43fa-91cc-f95db9196fc9" containerID="5173f5474ca753af0995474f2566a65993b9baf42047cfd39c59596fcc8b1a4e" exitCode=0 Oct 29 23:11:45.374394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.373965 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" event={"ID":"e53b6ac4-e35b-43fa-91cc-f95db9196fc9","Type":"ContainerDied","Data":"5173f5474ca753af0995474f2566a65993b9baf42047cfd39c59596fcc8b1a4e"} Oct 29 23:11:45.561847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.561814 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:45.645419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.645314 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-xks-ca\") pod \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " Oct 29 23:11:45.645419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.645408 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fdcf95e-b463-494d-8ebc-5b04b812fb19-kube-api-access-zrjdx\" (UniqueName: \"kubernetes.io/projected/2fdcf95e-b463-494d-8ebc-5b04b812fb19-kube-api-access-zrjdx\") pod \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " Oct 29 23:11:45.645719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.645657 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-openshift-ca\") pod \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\" (UID: \"2fdcf95e-b463-494d-8ebc-5b04b812fb19\") " Oct 29 23:11:45.646732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.646703 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-openshift-ca" pod "2fdcf95e-b463-494d-8ebc-5b04b812fb19" (UID: "2fdcf95e-b463-494d-8ebc-5b04b812fb19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:45.653063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.651094 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdcf95e-b463-494d-8ebc-5b04b812fb19-kube-api-access-zrjdx" pod "2fdcf95e-b463-494d-8ebc-5b04b812fb19" (UID: "2fdcf95e-b463-494d-8ebc-5b04b812fb19"). InnerVolumeSpecName "kube-api-access-zrjdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:45.653063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.652976 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-xks-ca" pod "2fdcf95e-b463-494d-8ebc-5b04b812fb19" (UID: "2fdcf95e-b463-494d-8ebc-5b04b812fb19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:45.747642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.747604 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrjdx\" (UniqueName: \"kubernetes.io/projected/2fdcf95e-b463-494d-8ebc-5b04b812fb19-kube-api-access-zrjdx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:45.747642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.747633 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:45.747642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:45.747644 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdcf95e-b463-494d-8ebc-5b04b812fb19-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:46.385301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.385263 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" Oct 29 23:11:46.385301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.385294 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj" event={"ID":"2fdcf95e-b463-494d-8ebc-5b04b812fb19","Type":"ContainerDied","Data":"6f487dfd671360a3c0cf9bd6ce91126f3846f7754dd8a5e76186acc129c34287"} Oct 29 23:11:46.385980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.385373 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f487dfd671360a3c0cf9bd6ce91126f3846f7754dd8a5e76186acc129c34287" Oct 29 23:11:46.545367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.545337 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:46.660869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.660778 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-xks-ca\") pod \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " Oct 29 23:11:46.661152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.660909 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-kube-api-access-d7fks\" (UniqueName: \"kubernetes.io/projected/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-kube-api-access-d7fks\") pod \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " Oct 29 23:11:46.661152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.660966 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-openshift-ca\") pod \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\" (UID: \"e53b6ac4-e35b-43fa-91cc-f95db9196fc9\") " Oct 29 23:11:46.661354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.661321 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-openshift-ca" pod "e53b6ac4-e35b-43fa-91cc-f95db9196fc9" (UID: "e53b6ac4-e35b-43fa-91cc-f95db9196fc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:46.661710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.661687 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:46.663067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.663041 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-xks-ca" pod "e53b6ac4-e35b-43fa-91cc-f95db9196fc9" (UID: "e53b6ac4-e35b-43fa-91cc-f95db9196fc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:46.663284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.663256 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-kube-api-access-d7fks" pod "e53b6ac4-e35b-43fa-91cc-f95db9196fc9" (UID: "e53b6ac4-e35b-43fa-91cc-f95db9196fc9"). InnerVolumeSpecName "kube-api-access-d7fks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:46.763506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.763471 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:46.763506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:46.763503 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7fks\" (UniqueName: \"kubernetes.io/projected/e53b6ac4-e35b-43fa-91cc-f95db9196fc9-kube-api-access-d7fks\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:11:47.394874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:47.394794 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" Oct 29 23:11:47.394874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:47.394831 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5" event={"ID":"e53b6ac4-e35b-43fa-91cc-f95db9196fc9","Type":"ContainerDied","Data":"dfb8fba5f08de3f20b2321bbfcfe8d621cd23b42a9a154d8334840be12198ca1"} Oct 29 23:11:47.395297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:47.394881 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfb8fba5f08de3f20b2321bbfcfe8d621cd23b42a9a154d8334840be12198ca1" Oct 29 23:11:55.947209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:55.947173 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp"] Oct 29 23:11:55.947763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:55.947747 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e53b6ac4-e35b-43fa-91cc-f95db9196fc9" containerName="revoke-keys" Oct 29 23:11:55.947763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:55.947763 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53b6ac4-e35b-43fa-91cc-f95db9196fc9" containerName="revoke-keys" Oct 29 23:11:55.947890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:55.947779 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2fdcf95e-b463-494d-8ebc-5b04b812fb19" containerName="revoke-keys" Oct 29 23:11:55.947890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:55.947784 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fdcf95e-b463-494d-8ebc-5b04b812fb19" containerName="revoke-keys" Oct 29 23:11:55.948011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:55.947893 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53b6ac4-e35b-43fa-91cc-f95db9196fc9" containerName="revoke-keys" Oct 29 23:11:55.948011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:55.947902 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fdcf95e-b463-494d-8ebc-5b04b812fb19" containerName="revoke-keys" Oct 29 23:11:55.951673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:55.951652 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:55.992881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:55.992843 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp"] Oct 29 23:11:56.024976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.024945 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:11:56.025632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.025438 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:11:56.041039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.040982 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mb7\" (UniqueName: \"kubernetes.io/projected/632ff5ad-1de4-4dce-b771-bef6c4b28731-kube-api-access-d7mb7\") pod \"maas-api-revoke-sub-f09f7f54-b6pxp\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.041322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.041299 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09f7f54-b6pxp\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.041572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.041551 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09f7f54-b6pxp\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.142930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.142883 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mb7\" (UniqueName: \"kubernetes.io/projected/632ff5ad-1de4-4dce-b771-bef6c4b28731-kube-api-access-d7mb7\") pod \"maas-api-revoke-sub-f09f7f54-b6pxp\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.143142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.142944 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09f7f54-b6pxp\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.143142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.143019 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09f7f54-b6pxp\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.143798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.143771 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09f7f54-b6pxp\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.145497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.145465 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09f7f54-b6pxp\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.167148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.167122 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mb7\" (UniqueName: \"kubernetes.io/projected/632ff5ad-1de4-4dce-b771-bef6c4b28731-kube-api-access-d7mb7\") pod \"maas-api-revoke-sub-f09f7f54-b6pxp\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.299297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.299257 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:11:56.614643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:56.614609 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp"] Oct 29 23:11:56.617198 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:56.617165 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632ff5ad_1de4_4dce_b771_bef6c4b28731.slice/crio-1ac45370af47d48bf690fc0d5e3479f735299d76d9ec915a590a9798c74358d6 WatchSource:0}: Error finding container 1ac45370af47d48bf690fc0d5e3479f735299d76d9ec915a590a9798c74358d6: Status 404 returned error can't find the container with id 1ac45370af47d48bf690fc0d5e3479f735299d76d9ec915a590a9798c74358d6 Oct 29 23:11:56.826371 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:11:56.826330 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:11:57.499946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:57.499900 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" event={"ID":"632ff5ad-1de4-4dce-b771-bef6c4b28731","Type":"ContainerStarted","Data":"7519e94541f6b6aecb7159799b421d380c0eaa926e7434ae92a5cdf04a61bd3d"} Oct 29 23:11:57.500550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:57.499968 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" event={"ID":"632ff5ad-1de4-4dce-b771-bef6c4b28731","Type":"ContainerStarted","Data":"1ac45370af47d48bf690fc0d5e3479f735299d76d9ec915a590a9798c74358d6"} Oct 29 23:11:57.541692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:57.541642 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" podStartSLOduration=2.5416275329999998 podStartE2EDuration="2.541627533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:57.540987144 +0000 UTC m=+1714.656906751" watchObservedRunningTime="2026-10-29 23:11:57.541627533 +0000 UTC m=+1714.657547139" Oct 29 23:11:58.288915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.288868 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn"] Oct 29 23:11:58.293672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.293647 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.313918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.313890 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn"] Oct 29 23:11:58.391287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.391255 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381f1999-lwznn\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.391456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.391332 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9c8c\" (UniqueName: \"kubernetes.io/projected/35acf06c-b040-4178-88dd-d48cc6d6291d-kube-api-access-r9c8c\") pod \"maas-api-revoke-sub-381f1999-lwznn\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.391456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.391396 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381f1999-lwznn\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.492794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.492762 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381f1999-lwznn\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.492981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.492821 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9c8c\" (UniqueName: \"kubernetes.io/projected/35acf06c-b040-4178-88dd-d48cc6d6291d-kube-api-access-r9c8c\") pod \"maas-api-revoke-sub-381f1999-lwznn\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.492981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.492873 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381f1999-lwznn\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.493506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.493484 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381f1999-lwznn\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.495252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.495231 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381f1999-lwznn\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.509801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.509769 5736 generic.go:356] "Generic (PLEG): container finished" podID="632ff5ad-1de4-4dce-b771-bef6c4b28731" containerID="7519e94541f6b6aecb7159799b421d380c0eaa926e7434ae92a5cdf04a61bd3d" exitCode=0 Oct 29 23:11:58.510189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.509848 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" event={"ID":"632ff5ad-1de4-4dce-b771-bef6c4b28731","Type":"ContainerDied","Data":"7519e94541f6b6aecb7159799b421d380c0eaa926e7434ae92a5cdf04a61bd3d"} Oct 29 23:11:58.516081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.516051 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9c8c\" (UniqueName: \"kubernetes.io/projected/35acf06c-b040-4178-88dd-d48cc6d6291d-kube-api-access-r9c8c\") pod \"maas-api-revoke-sub-381f1999-lwznn\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.608073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.607988 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:11:58.919044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:58.919012 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn"] Oct 29 23:11:58.935960 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:11:58.935693 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35acf06c_b040_4178_88dd_d48cc6d6291d.slice/crio-d053b19255ce0098a1fdc1f4bed162d7f0d891bfb091755accaa1202ac71eb83 WatchSource:0}: Error finding container d053b19255ce0098a1fdc1f4bed162d7f0d891bfb091755accaa1202ac71eb83: Status 404 returned error can't find the container with id d053b19255ce0098a1fdc1f4bed162d7f0d891bfb091755accaa1202ac71eb83 Oct 29 23:11:59.530658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:59.528400 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" event={"ID":"35acf06c-b040-4178-88dd-d48cc6d6291d","Type":"ContainerStarted","Data":"6b4023b64d36913aeccdf606b7a38c7e9fcb9e9f8b5c5b08b5af9ef2282dd58f"} Oct 29 23:11:59.530658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:59.528480 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" event={"ID":"35acf06c-b040-4178-88dd-d48cc6d6291d","Type":"ContainerStarted","Data":"d053b19255ce0098a1fdc1f4bed162d7f0d891bfb091755accaa1202ac71eb83"} Oct 29 23:11:59.631660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:11:59.612878 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" podStartSLOduration=1.612857904 podStartE2EDuration="1.612857904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:59.572725068 +0000 UTC m=+1716.688644674" watchObservedRunningTime="2026-10-29 23:11:59.612857904 +0000 UTC m=+1716.728777511" Oct 29 23:12:00.392656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.384791 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:12:00.647667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.646555 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632ff5ad-1de4-4dce-b771-bef6c4b28731-kube-api-access-d7mb7\" (UniqueName: \"kubernetes.io/projected/632ff5ad-1de4-4dce-b771-bef6c4b28731-kube-api-access-d7mb7\") pod \"632ff5ad-1de4-4dce-b771-bef6c4b28731\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " Oct 29 23:12:00.647667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.646787 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-openshift-ca\") pod \"632ff5ad-1de4-4dce-b771-bef6c4b28731\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " Oct 29 23:12:00.647667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.646831 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-xks-ca\") pod \"632ff5ad-1de4-4dce-b771-bef6c4b28731\" (UID: \"632ff5ad-1de4-4dce-b771-bef6c4b28731\") " Oct 29 23:12:00.757853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.710069 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-openshift-ca" pod "632ff5ad-1de4-4dce-b771-bef6c4b28731" (UID: "632ff5ad-1de4-4dce-b771-bef6c4b28731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:00.831603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.831539 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:00.954974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.954887 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632ff5ad-1de4-4dce-b771-bef6c4b28731-kube-api-access-d7mb7" pod "632ff5ad-1de4-4dce-b771-bef6c4b28731" (UID: "632ff5ad-1de4-4dce-b771-bef6c4b28731"). InnerVolumeSpecName "kube-api-access-d7mb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:00.956682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.955392 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-xks-ca" pod "632ff5ad-1de4-4dce-b771-bef6c4b28731" (UID: "632ff5ad-1de4-4dce-b771-bef6c4b28731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:12:00.975179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.975145 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" event={"ID":"632ff5ad-1de4-4dce-b771-bef6c4b28731","Type":"ContainerDied","Data":"1ac45370af47d48bf690fc0d5e3479f735299d76d9ec915a590a9798c74358d6"} Oct 29 23:12:00.996953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.996916 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ac45370af47d48bf690fc0d5e3479f735299d76d9ec915a590a9798c74358d6" Oct 29 23:12:00.997616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.996337 5736 generic.go:356] "Generic (PLEG): container finished" podID="35acf06c-b040-4178-88dd-d48cc6d6291d" containerID="6b4023b64d36913aeccdf606b7a38c7e9fcb9e9f8b5c5b08b5af9ef2282dd58f" exitCode=0 Oct 29 23:12:01.010078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:00.975409 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp" Oct 29 23:12:01.017683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.017647 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.036796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.036759 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7mb7\" (UniqueName: \"kubernetes.io/projected/632ff5ad-1de4-4dce-b771-bef6c4b28731-kube-api-access-d7mb7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:01.037024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.037004 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632ff5ad-1de4-4dce-b771-bef6c4b28731-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:01.148606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.146697 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.154040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.153973 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.259976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.259926 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.260722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.260691 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.288604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.263677 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.289166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.281484 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 23:12:01.290944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.290915 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 29 23:12:01.362604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.355002 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" event={"ID":"35acf06c-b040-4178-88dd-d48cc6d6291d","Type":"ContainerDied","Data":"6b4023b64d36913aeccdf606b7a38c7e9fcb9e9f8b5c5b08b5af9ef2282dd58f"} Oct 29 23:12:01.421669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.417420 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.493600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.492924 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.701820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.701705 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.714776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.714723 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.716030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.716001 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.716555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.716200 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 23:12:01.716728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.716225 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 29 23:12:01.716982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.716651 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.717373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.716677 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 29 23:12:01.736634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.734766 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.852614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.850937 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:01.852614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.850997 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:01.903616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:01.901908 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.022617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.007701 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:02.022617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.007806 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.182666 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.167678 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.195617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.185169 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:12:02.269851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.269807 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.284668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.270424 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 29 23:12:02.285262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.276750 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 23:12:02.285741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.285718 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:12:02.285907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.285895 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 23:12:02.304601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.291211 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.304601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.293033 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.304601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.293358 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:12:02.345605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.319679 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 29 23:12:02.345605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.320145 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 23:12:02.508611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.493696 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 23:12:02.508611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.494170 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:12:02.508611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.502549 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.508611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.502966 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 23:12:02.626881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.626780 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.647033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.647003 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:12:02.705628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.703742 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.722615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.722550 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.725475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.722882 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.733989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.722918 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 23:12:02.735409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.722947 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.735409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.724258 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:12:02.735409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.734499 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 23:12:02.735409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.734542 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:12:02.735409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.734598 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 23:12:02.735409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.734633 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:12:02.736767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.724305 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.751785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.751741 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:02.930663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.926719 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 29 23:12:03.005764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.996002 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:12:03.005764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.996068 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 23:12:03.005764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:02.996103 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 23:12:03.268632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.260148 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.314606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.289718 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.315106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.301071 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 23:12:03.315447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.302773 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 29 23:12:03.333602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.302808 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 23:12:03.349555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.302831 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 23:12:03.349555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.325208 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.610658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.498807 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.610658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.514794 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.627615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.516698 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:03.648701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.631674 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.650718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.648023 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.732759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.732705 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.755612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.735234 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 23:12:03.758656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.758616 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.784607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.759530 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:03.785312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.785269 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 29 23:12:03.785795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:03.785771 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 23:12:04.067942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.067898 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.242334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.081010 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 29 23:12:04.295606 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:12:04.146247 5736 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.14s" Oct 29 23:12:04.347606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.316519 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:12:04.347606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.315845 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.347606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.316017 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 23:12:04.347606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.316046 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 23:12:04.347606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.341878 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 23:12:04.425618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.424670 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.484326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.484293 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:04.514780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.514659 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.515024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.505691 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded" Oct 29 23:12:04.636593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.622188 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 29 23:12:04.640656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.504296 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.641416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.610471 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 23:12:04.641531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.641501 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 23:12:04.673605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.673556 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 23:12:04.779656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.778710 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.810797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.810748 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 23:12:04.865603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.551848 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.885994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.877963 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.899180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.897738 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 23:12:04.937640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.929142 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.937640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.929245 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 23:12:04.949619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.946560 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.968944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.968864 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:04.990277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.990238 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:12:04.992284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.992263 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 23:12:04.992461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.992450 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:12:04.992673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.992656 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 23:12:04.992923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.992907 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 23:12:04.993090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.993078 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:12:04.993233 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.993221 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:12:04.997206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:04.994391 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 23:12:05.039188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.034573 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:12:05.046085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.046053 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:12:05.046242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.046135 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:12:05.046639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.046619 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:12:05.046700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.046663 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" event={"ID":"35acf06c-b040-4178-88dd-d48cc6d6291d","Type":"ContainerDied","Data":"d053b19255ce0098a1fdc1f4bed162d7f0d891bfb091755accaa1202ac71eb83"} Oct 29 23:12:05.046756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.046713 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d053b19255ce0098a1fdc1f4bed162d7f0d891bfb091755accaa1202ac71eb83" Oct 29 23:12:05.056046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.056016 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:12:05.056306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.056294 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:12:05.056507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.056483 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:12:05.058279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.058250 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 23:12:05.201782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.201747 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-xks-ca\") pod \"35acf06c-b040-4178-88dd-d48cc6d6291d\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " Oct 29 23:12:05.202232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.202211 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-openshift-ca\") pod \"35acf06c-b040-4178-88dd-d48cc6d6291d\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " Oct 29 23:12:05.202422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.202406 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35acf06c-b040-4178-88dd-d48cc6d6291d-kube-api-access-r9c8c\" (UniqueName: \"kubernetes.io/projected/35acf06c-b040-4178-88dd-d48cc6d6291d-kube-api-access-r9c8c\") pod \"35acf06c-b040-4178-88dd-d48cc6d6291d\" (UID: \"35acf06c-b040-4178-88dd-d48cc6d6291d\") " Oct 29 23:12:05.222751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.221159 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-openshift-ca" pod "35acf06c-b040-4178-88dd-d48cc6d6291d" (UID: "35acf06c-b040-4178-88dd-d48cc6d6291d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:05.231113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.226984 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-xks-ca" pod "35acf06c-b040-4178-88dd-d48cc6d6291d" (UID: "35acf06c-b040-4178-88dd-d48cc6d6291d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:12:05.231277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.231223 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35acf06c-b040-4178-88dd-d48cc6d6291d-kube-api-access-r9c8c" pod "35acf06c-b040-4178-88dd-d48cc6d6291d" (UID: "35acf06c-b040-4178-88dd-d48cc6d6291d"). InnerVolumeSpecName "kube-api-access-r9c8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:05.239941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.239918 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf"] Oct 29 23:12:05.240149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.240127 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" containerID="cri-o://cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c" gracePeriod=30 Oct 29 23:12:05.307676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.305354 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:05.307676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.305393 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acf06c-b040-4178-88dd-d48cc6d6291d-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:05.307676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:05.305408 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9c8c\" (UniqueName: \"kubernetes.io/projected/35acf06c-b040-4178-88dd-d48cc6d6291d-kube-api-access-r9c8c\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:06.029131 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:06.029087 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn" Oct 29 23:12:07.387038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.386990 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v"] Oct 29 23:12:07.387670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.387650 5736 cpu_manager.go:414] "containerMap: removing container" podUID="35acf06c-b040-4178-88dd-d48cc6d6291d" containerName="revoke-keys" Oct 29 23:12:07.387670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.387670 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35acf06c-b040-4178-88dd-d48cc6d6291d" containerName="revoke-keys" Oct 29 23:12:07.387796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.387725 5736 cpu_manager.go:414] "containerMap: removing container" podUID="632ff5ad-1de4-4dce-b771-bef6c4b28731" containerName="revoke-keys" Oct 29 23:12:07.387796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.387732 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632ff5ad-1de4-4dce-b771-bef6c4b28731" containerName="revoke-keys" Oct 29 23:12:07.387892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.387820 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="632ff5ad-1de4-4dce-b771-bef6c4b28731" containerName="revoke-keys" Oct 29 23:12:07.387892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.387831 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="35acf06c-b040-4178-88dd-d48cc6d6291d" containerName="revoke-keys" Oct 29 23:12:07.394896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.394876 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.411697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.411664 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v"] Oct 29 23:12:07.419882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.419857 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:12:07.419997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.419927 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:12:07.537414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.537378 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd96n\" (UniqueName: \"kubernetes.io/projected/0f64ca3e-213a-473b-8722-27255ecbd7ea-kube-api-access-gd96n\") pod \"maas-api-revoke-sub-a041ba13-44n8v\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.537414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.537415 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a041ba13-44n8v\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.537731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.537709 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a041ba13-44n8v\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.638572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.638491 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a041ba13-44n8v\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.638726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.638632 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd96n\" (UniqueName: \"kubernetes.io/projected/0f64ca3e-213a-473b-8722-27255ecbd7ea-kube-api-access-gd96n\") pod \"maas-api-revoke-sub-a041ba13-44n8v\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.638726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.638663 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a041ba13-44n8v\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.639333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.639307 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a041ba13-44n8v\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.641347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.641321 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a041ba13-44n8v\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.661937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.661914 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd96n\" (UniqueName: \"kubernetes.io/projected/0f64ca3e-213a-473b-8722-27255ecbd7ea-kube-api-access-gd96n\") pod \"maas-api-revoke-sub-a041ba13-44n8v\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.708273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.708249 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:07.807797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.807762 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw"] Oct 29 23:12:07.813459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.813431 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:07.834003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.833968 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw"] Oct 29 23:12:07.942956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.942884 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da536371-ef7f-438b-9124-185a7b439107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ef3bb3e-qw7lw\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:07.942956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.942927 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da536371-ef7f-438b-9124-185a7b439107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ef3bb3e-qw7lw\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:07.943137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:07.943100 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhmbm\" (UniqueName: \"kubernetes.io/projected/da536371-ef7f-438b-9124-185a7b439107-kube-api-access-mhmbm\") pod \"maas-api-revoke-sub-6ef3bb3e-qw7lw\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:08.038650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.038569 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v"] Oct 29 23:12:08.044569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.044538 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da536371-ef7f-438b-9124-185a7b439107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ef3bb3e-qw7lw\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:08.044787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.044639 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da536371-ef7f-438b-9124-185a7b439107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ef3bb3e-qw7lw\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:08.044976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.044913 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhmbm\" (UniqueName: \"kubernetes.io/projected/da536371-ef7f-438b-9124-185a7b439107-kube-api-access-mhmbm\") pod \"maas-api-revoke-sub-6ef3bb3e-qw7lw\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:08.045500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.045460 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da536371-ef7f-438b-9124-185a7b439107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ef3bb3e-qw7lw\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:08.048928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.048907 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da536371-ef7f-438b-9124-185a7b439107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ef3bb3e-qw7lw\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:08.055208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.055177 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" event={"ID":"0f64ca3e-213a-473b-8722-27255ecbd7ea","Type":"ContainerStarted","Data":"e9bdd0f0441834600d9cad555c8f38a2706dd5bc55b154b68b0cccd6cade2c65"} Oct 29 23:12:08.068802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.068780 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhmbm\" (UniqueName: \"kubernetes.io/projected/da536371-ef7f-438b-9124-185a7b439107-kube-api-access-mhmbm\") pod \"maas-api-revoke-sub-6ef3bb3e-qw7lw\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:08.137397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.137367 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:08.307127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.307087 5736 patch_prober.go:29] interesting pod/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:12:08.307296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.307143 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:12:08.443781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:08.443714 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw"] Oct 29 23:12:08.445514 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:12:08.445487 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda536371_ef7f_438b_9124_185a7b439107.slice/crio-cdcde0e5ec4416c609f6d54cc548c8cce7a4ae16cbbc9238da2860ac54ab998a WatchSource:0}: Error finding container cdcde0e5ec4416c609f6d54cc548c8cce7a4ae16cbbc9238da2860ac54ab998a: Status 404 returned error can't find the container with id cdcde0e5ec4416c609f6d54cc548c8cce7a4ae16cbbc9238da2860ac54ab998a Oct 29 23:12:09.065667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:09.065562 5736 generic.go:356] "Generic (PLEG): container finished" podID="0f64ca3e-213a-473b-8722-27255ecbd7ea" containerID="e702b06d2a4b960aa0874347ee28967a8a74ffda82ac8668a996ff68bc209978" exitCode=0 Oct 29 23:12:09.065667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:09.065615 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" event={"ID":"0f64ca3e-213a-473b-8722-27255ecbd7ea","Type":"ContainerDied","Data":"e702b06d2a4b960aa0874347ee28967a8a74ffda82ac8668a996ff68bc209978"} Oct 29 23:12:09.067538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:09.067511 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" event={"ID":"da536371-ef7f-438b-9124-185a7b439107","Type":"ContainerStarted","Data":"790b20112870ae62ce27ebf24557232a0b1f352cec22a1c2066bfab0b9268275"} Oct 29 23:12:09.067697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:09.067557 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" event={"ID":"da536371-ef7f-438b-9124-185a7b439107","Type":"ContainerStarted","Data":"cdcde0e5ec4416c609f6d54cc548c8cce7a4ae16cbbc9238da2860ac54ab998a"} Oct 29 23:12:09.147954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:09.147898 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" podStartSLOduration=2.1478831 podStartE2EDuration="2.1478831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:09.145020415 +0000 UTC m=+1726.260940018" watchObservedRunningTime="2026-10-29 23:12:09.1478831 +0000 UTC m=+1726.263802746" Oct 29 23:12:10.083449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.083396 5736 generic.go:356] "Generic (PLEG): container finished" podID="da536371-ef7f-438b-9124-185a7b439107" containerID="790b20112870ae62ce27ebf24557232a0b1f352cec22a1c2066bfab0b9268275" exitCode=0 Oct 29 23:12:10.084293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.084260 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" event={"ID":"da536371-ef7f-438b-9124-185a7b439107","Type":"ContainerDied","Data":"790b20112870ae62ce27ebf24557232a0b1f352cec22a1c2066bfab0b9268275"} Oct 29 23:12:10.369381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.369352 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:10.411247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.411208 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-openshift-ca\") pod \"0f64ca3e-213a-473b-8722-27255ecbd7ea\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " Oct 29 23:12:10.411433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.411411 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f64ca3e-213a-473b-8722-27255ecbd7ea-kube-api-access-gd96n\" (UniqueName: \"kubernetes.io/projected/0f64ca3e-213a-473b-8722-27255ecbd7ea-kube-api-access-gd96n\") pod \"0f64ca3e-213a-473b-8722-27255ecbd7ea\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " Oct 29 23:12:10.411617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.411574 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-xks-ca\") pod \"0f64ca3e-213a-473b-8722-27255ecbd7ea\" (UID: \"0f64ca3e-213a-473b-8722-27255ecbd7ea\") " Oct 29 23:12:10.418186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.417935 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-openshift-ca" pod "0f64ca3e-213a-473b-8722-27255ecbd7ea" (UID: "0f64ca3e-213a-473b-8722-27255ecbd7ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:10.419970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.419895 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-xks-ca" pod "0f64ca3e-213a-473b-8722-27255ecbd7ea" (UID: "0f64ca3e-213a-473b-8722-27255ecbd7ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:12:10.423068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.423022 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f64ca3e-213a-473b-8722-27255ecbd7ea-kube-api-access-gd96n" pod "0f64ca3e-213a-473b-8722-27255ecbd7ea" (UID: "0f64ca3e-213a-473b-8722-27255ecbd7ea"). InnerVolumeSpecName "kube-api-access-gd96n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:10.485153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.485126 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:12:10.513642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.513610 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd96n\" (UniqueName: \"kubernetes.io/projected/0f64ca3e-213a-473b-8722-27255ecbd7ea-kube-api-access-gd96n\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.513642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.513638 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.513822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.513654 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ca3e-213a-473b-8722-27255ecbd7ea-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.617418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617379 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-certs\") pod \"34a44adc-0ef9-45aa-a229-1857e9617cad\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " Oct 29 23:12:10.617675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617444 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-kube-api-access-bm54s\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-kube-api-access-bm54s\") pod \"34a44adc-0ef9-45aa-a229-1857e9617cad\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " Oct 29 23:12:10.617675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617475 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-envoy\") pod \"34a44adc-0ef9-45aa-a229-1857e9617cad\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " Oct 29 23:12:10.617675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617537 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-data\") pod \"34a44adc-0ef9-45aa-a229-1857e9617cad\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " Oct 29 23:12:10.617675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617607 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-podinfo\") pod \"34a44adc-0ef9-45aa-a229-1857e9617cad\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " Oct 29 23:12:10.617935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617730 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-credential-socket\") pod \"34a44adc-0ef9-45aa-a229-1857e9617cad\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " Oct 29 23:12:10.617935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617796 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-token\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-token\") pod \"34a44adc-0ef9-45aa-a229-1857e9617cad\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " Oct 29 23:12:10.617935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617823 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-certs" pod "34a44adc-0ef9-45aa-a229-1857e9617cad" (UID: "34a44adc-0ef9-45aa-a229-1857e9617cad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.617935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617918 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a44adc-0ef9-45aa-a229-1857e9617cad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a44adc-0ef9-45aa-a229-1857e9617cad-istiod-ca-cert\") pod \"34a44adc-0ef9-45aa-a229-1857e9617cad\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " Oct 29 23:12:10.618138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.618025 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-socket\") pod \"34a44adc-0ef9-45aa-a229-1857e9617cad\" (UID: \"34a44adc-0ef9-45aa-a229-1857e9617cad\") " Oct 29 23:12:10.618138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.617995 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-data" pod "34a44adc-0ef9-45aa-a229-1857e9617cad" (UID: "34a44adc-0ef9-45aa-a229-1857e9617cad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.618138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.618041 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-credential-socket" pod "34a44adc-0ef9-45aa-a229-1857e9617cad" (UID: "34a44adc-0ef9-45aa-a229-1857e9617cad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.618308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.618270 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-socket" pod "34a44adc-0ef9-45aa-a229-1857e9617cad" (UID: "34a44adc-0ef9-45aa-a229-1857e9617cad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.618482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.618374 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a44adc-0ef9-45aa-a229-1857e9617cad-istiod-ca-cert" pod "34a44adc-0ef9-45aa-a229-1857e9617cad" (UID: "34a44adc-0ef9-45aa-a229-1857e9617cad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:10.618854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.618830 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.618963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.618859 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.618963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.618875 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a44adc-0ef9-45aa-a229-1857e9617cad-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.618963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.618888 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.618963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.618900 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.620661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.620372 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-kube-api-access-bm54s" pod "34a44adc-0ef9-45aa-a229-1857e9617cad" (UID: "34a44adc-0ef9-45aa-a229-1857e9617cad"). InnerVolumeSpecName "kube-api-access-bm54s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:10.620902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.620882 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-envoy" pod "34a44adc-0ef9-45aa-a229-1857e9617cad" (UID: "34a44adc-0ef9-45aa-a229-1857e9617cad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:12:10.621033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.621014 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-podinfo" pod "34a44adc-0ef9-45aa-a229-1857e9617cad" (UID: "34a44adc-0ef9-45aa-a229-1857e9617cad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:12:10.621365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.621346 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-token" pod "34a44adc-0ef9-45aa-a229-1857e9617cad" (UID: "34a44adc-0ef9-45aa-a229-1857e9617cad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:10.629374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.628223 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7"] Oct 29 23:12:10.633323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.633301 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f2c8d3-5zrc7"] Oct 29 23:12:10.720796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.720759 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm54s\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-kube-api-access-bm54s\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.720796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.720798 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.721008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.720816 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.721008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.720829 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a44adc-0ef9-45aa-a229-1857e9617cad-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:10.753203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.753123 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8"] Oct 29 23:12:10.753797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.753779 5736 cpu_manager.go:414] "containerMap: removing container" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" Oct 29 23:12:10.753797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.753797 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" Oct 29 23:12:10.753944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.753819 5736 cpu_manager.go:414] "containerMap: removing container" podUID="0f64ca3e-213a-473b-8722-27255ecbd7ea" containerName="revoke-keys" Oct 29 23:12:10.753944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.753825 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f64ca3e-213a-473b-8722-27255ecbd7ea" containerName="revoke-keys" Oct 29 23:12:10.754011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.753947 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerName="istio-proxy" Oct 29 23:12:10.754011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.753958 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f64ca3e-213a-473b-8722-27255ecbd7ea" containerName="revoke-keys" Oct 29 23:12:10.757746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.757727 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:10.777312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.777013 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8"] Oct 29 23:12:10.830793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.830716 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:10.831097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.831069 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:10.831450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.831429 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69kgf\" (UniqueName: \"kubernetes.io/projected/63d0207d-7a96-4e74-bdb8-b3c2de58b219-kube-api-access-69kgf\") pod \"maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:10.932841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.932802 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:10.933012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.932871 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:10.933102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.933039 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69kgf\" (UniqueName: \"kubernetes.io/projected/63d0207d-7a96-4e74-bdb8-b3c2de58b219-kube-api-access-69kgf\") pod \"maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:10.933680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.933651 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:10.935354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.935332 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:10.956527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:10.956503 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69kgf\" (UniqueName: \"kubernetes.io/projected/63d0207d-7a96-4e74-bdb8-b3c2de58b219-kube-api-access-69kgf\") pod \"maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:11.008380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.008297 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e539e5-7b25-449f-bfcd-122b9b3642ab" path="/var/lib/kubelet/pods/e1e539e5-7b25-449f-bfcd-122b9b3642ab/volumes" Oct 29 23:12:11.071636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.071611 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:11.095820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.095790 5736 generic.go:356] "Generic (PLEG): container finished" podID="34a44adc-0ef9-45aa-a229-1857e9617cad" containerID="cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c" exitCode=0 Oct 29 23:12:11.096227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.095860 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" Oct 29 23:12:11.096227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.096027 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" event={"ID":"34a44adc-0ef9-45aa-a229-1857e9617cad","Type":"ContainerDied","Data":"cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c"} Oct 29 23:12:11.096227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.096083 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf" event={"ID":"34a44adc-0ef9-45aa-a229-1857e9617cad","Type":"ContainerDied","Data":"d8b95e303976899999844decc30fd4f325236b4b6440e1aacfef5babf3d89e61"} Oct 29 23:12:11.096227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.096127 5736 scope.go:122] "RemoveContainer" containerID="cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c" Oct 29 23:12:11.098998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.098310 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" Oct 29 23:12:11.098998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.098923 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v" event={"ID":"0f64ca3e-213a-473b-8722-27255ecbd7ea","Type":"ContainerDied","Data":"e9bdd0f0441834600d9cad555c8f38a2706dd5bc55b154b68b0cccd6cade2c65"} Oct 29 23:12:11.098998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.098962 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9bdd0f0441834600d9cad555c8f38a2706dd5bc55b154b68b0cccd6cade2c65" Oct 29 23:12:11.113292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.113261 5736 scope.go:122] "RemoveContainer" containerID="cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c" Oct 29 23:12:11.114543 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:12:11.114518 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c\": container with ID starting with cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c not found: ID does not exist" containerID="cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c" Oct 29 23:12:11.114692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.114550 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c"} err="failed to get container status \"cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c\": rpc error: code = NotFound desc = could not find container \"cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c\": container with ID starting with cadf965b04cc3348cbd67f1e2b1c31edc5a8e9e395192cea61c57a30915bcc5c not found: ID does not exist" Oct 29 23:12:11.170390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.168661 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf"] Oct 29 23:12:11.174559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.174530 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7af1a5-openshift-default-f86877477-m7rqf"] Oct 29 23:12:11.307221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.307191 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:11.389488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.389460 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8"] Oct 29 23:12:11.393213 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:12:11.393132 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d0207d_7a96_4e74_bdb8_b3c2de58b219.slice/crio-1523a75bf2026fde04aff0a2386a4dcd4a51f073ab0a45ad6164802ebf4dfb15 WatchSource:0}: Error finding container 1523a75bf2026fde04aff0a2386a4dcd4a51f073ab0a45ad6164802ebf4dfb15: Status 404 returned error can't find the container with id 1523a75bf2026fde04aff0a2386a4dcd4a51f073ab0a45ad6164802ebf4dfb15 Oct 29 23:12:11.441246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.441213 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da536371-ef7f-438b-9124-185a7b439107-kube-api-access-mhmbm\" (UniqueName: \"kubernetes.io/projected/da536371-ef7f-438b-9124-185a7b439107-kube-api-access-mhmbm\") pod \"da536371-ef7f-438b-9124-185a7b439107\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " Oct 29 23:12:11.441376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.441262 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da536371-ef7f-438b-9124-185a7b439107-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da536371-ef7f-438b-9124-185a7b439107-maas-api-openshift-ca\") pod \"da536371-ef7f-438b-9124-185a7b439107\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " Oct 29 23:12:11.441463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.441410 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da536371-ef7f-438b-9124-185a7b439107-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da536371-ef7f-438b-9124-185a7b439107-maas-api-xks-ca\") pod \"da536371-ef7f-438b-9124-185a7b439107\" (UID: \"da536371-ef7f-438b-9124-185a7b439107\") " Oct 29 23:12:11.441974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.441905 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da536371-ef7f-438b-9124-185a7b439107-maas-api-openshift-ca" pod "da536371-ef7f-438b-9124-185a7b439107" (UID: "da536371-ef7f-438b-9124-185a7b439107"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:11.442342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.442302 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da536371-ef7f-438b-9124-185a7b439107-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:11.444078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.444055 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da536371-ef7f-438b-9124-185a7b439107-kube-api-access-mhmbm" pod "da536371-ef7f-438b-9124-185a7b439107" (UID: "da536371-ef7f-438b-9124-185a7b439107"). InnerVolumeSpecName "kube-api-access-mhmbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:11.444234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.444212 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da536371-ef7f-438b-9124-185a7b439107-maas-api-xks-ca" pod "da536371-ef7f-438b-9124-185a7b439107" (UID: "da536371-ef7f-438b-9124-185a7b439107"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:12:11.544091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.544015 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da536371-ef7f-438b-9124-185a7b439107-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:11.544091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:11.544047 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhmbm\" (UniqueName: \"kubernetes.io/projected/da536371-ef7f-438b-9124-185a7b439107-kube-api-access-mhmbm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:12.109300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:12.109207 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" Oct 29 23:12:12.109300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:12.109213 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw" event={"ID":"da536371-ef7f-438b-9124-185a7b439107","Type":"ContainerDied","Data":"cdcde0e5ec4416c609f6d54cc548c8cce7a4ae16cbbc9238da2860ac54ab998a"} Oct 29 23:12:12.109300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:12.109270 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdcde0e5ec4416c609f6d54cc548c8cce7a4ae16cbbc9238da2860ac54ab998a" Oct 29 23:12:12.112645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:12.112615 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" event={"ID":"63d0207d-7a96-4e74-bdb8-b3c2de58b219","Type":"ContainerStarted","Data":"ce9f9a9155587c79a44e61fb30f269222a2481fce29e9913ce19ee45a653aaaf"} Oct 29 23:12:12.112809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:12.112666 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" event={"ID":"63d0207d-7a96-4e74-bdb8-b3c2de58b219","Type":"ContainerStarted","Data":"1523a75bf2026fde04aff0a2386a4dcd4a51f073ab0a45ad6164802ebf4dfb15"} Oct 29 23:12:12.155884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:12.155822 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" podStartSLOduration=2.155804923 podStartE2EDuration="2.155804923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:12.152769291 +0000 UTC m=+1729.268688897" watchObservedRunningTime="2026-10-29 23:12:12.155804923 +0000 UTC m=+1729.271724530" Oct 29 23:12:13.005806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:13.005774 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a44adc-0ef9-45aa-a229-1857e9617cad" path="/var/lib/kubelet/pods/34a44adc-0ef9-45aa-a229-1857e9617cad/volumes" Oct 29 23:12:13.122880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:13.122840 5736 generic.go:356] "Generic (PLEG): container finished" podID="63d0207d-7a96-4e74-bdb8-b3c2de58b219" containerID="ce9f9a9155587c79a44e61fb30f269222a2481fce29e9913ce19ee45a653aaaf" exitCode=0 Oct 29 23:12:13.123250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:13.122894 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" event={"ID":"63d0207d-7a96-4e74-bdb8-b3c2de58b219","Type":"ContainerDied","Data":"ce9f9a9155587c79a44e61fb30f269222a2481fce29e9913ce19ee45a653aaaf"} Oct 29 23:12:14.287693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.287666 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:14.384054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.383966 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-openshift-ca\") pod \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " Oct 29 23:12:14.384054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.384043 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-xks-ca\") pod \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " Oct 29 23:12:14.384263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.384245 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d0207d-7a96-4e74-bdb8-b3c2de58b219-kube-api-access-69kgf\" (UniqueName: \"kubernetes.io/projected/63d0207d-7a96-4e74-bdb8-b3c2de58b219-kube-api-access-69kgf\") pod \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\" (UID: \"63d0207d-7a96-4e74-bdb8-b3c2de58b219\") " Oct 29 23:12:14.384353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.384324 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-openshift-ca" pod "63d0207d-7a96-4e74-bdb8-b3c2de58b219" (UID: "63d0207d-7a96-4e74-bdb8-b3c2de58b219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:14.384861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.384840 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:14.386285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.386264 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d0207d-7a96-4e74-bdb8-b3c2de58b219-kube-api-access-69kgf" pod "63d0207d-7a96-4e74-bdb8-b3c2de58b219" (UID: "63d0207d-7a96-4e74-bdb8-b3c2de58b219"). InnerVolumeSpecName "kube-api-access-69kgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:14.386347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.386264 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-xks-ca" pod "63d0207d-7a96-4e74-bdb8-b3c2de58b219" (UID: "63d0207d-7a96-4e74-bdb8-b3c2de58b219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:12:14.486178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.486132 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69kgf\" (UniqueName: \"kubernetes.io/projected/63d0207d-7a96-4e74-bdb8-b3c2de58b219-kube-api-access-69kgf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:14.486178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:14.486176 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d0207d-7a96-4e74-bdb8-b3c2de58b219-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:15.141791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:15.141763 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" Oct 29 23:12:15.141936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:15.141803 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8" event={"ID":"63d0207d-7a96-4e74-bdb8-b3c2de58b219","Type":"ContainerDied","Data":"1523a75bf2026fde04aff0a2386a4dcd4a51f073ab0a45ad6164802ebf4dfb15"} Oct 29 23:12:15.141936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:15.141857 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1523a75bf2026fde04aff0a2386a4dcd4a51f073ab0a45ad6164802ebf4dfb15" Oct 29 23:12:18.716269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.716234 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g"] Oct 29 23:12:18.716818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.716803 5736 cpu_manager.go:414] "containerMap: removing container" podUID="da536371-ef7f-438b-9124-185a7b439107" containerName="revoke-keys" Oct 29 23:12:18.716893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.716819 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da536371-ef7f-438b-9124-185a7b439107" containerName="revoke-keys" Oct 29 23:12:18.716893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.716857 5736 cpu_manager.go:414] "containerMap: removing container" podUID="63d0207d-7a96-4e74-bdb8-b3c2de58b219" containerName="revoke-keys" Oct 29 23:12:18.716893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.716862 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d0207d-7a96-4e74-bdb8-b3c2de58b219" containerName="revoke-keys" Oct 29 23:12:18.717068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.716959 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d0207d-7a96-4e74-bdb8-b3c2de58b219" containerName="revoke-keys" Oct 29 23:12:18.717068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.716972 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="da536371-ef7f-438b-9124-185a7b439107" containerName="revoke-keys" Oct 29 23:12:18.720908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.720890 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:18.741521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.741493 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g"] Oct 29 23:12:18.768324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.768292 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:12:18.768458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.768430 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:12:18.881565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.881531 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:18.881764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.881688 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:18.881842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.881808 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tsx\" (UniqueName: \"kubernetes.io/projected/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-kube-api-access-b8tsx\") pod \"maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:18.983009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.982933 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:18.983162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.983037 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:18.983162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.983088 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tsx\" (UniqueName: \"kubernetes.io/projected/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-kube-api-access-b8tsx\") pod \"maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:18.983637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.983595 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:18.985394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:18.985371 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:19.006152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:19.006122 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tsx\" (UniqueName: \"kubernetes.io/projected/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-kube-api-access-b8tsx\") pod \"maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:19.036649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:19.036625 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:19.346309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:19.346276 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g"] Oct 29 23:12:19.348450 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:12:19.348412 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a073fb_abcb_41fe_a6e7_72d6fedc5c8d.slice/crio-d424f987ea581a2ffe9951e358b03b220b71c2080f00e4229b0fe1fb04148996 WatchSource:0}: Error finding container d424f987ea581a2ffe9951e358b03b220b71c2080f00e4229b0fe1fb04148996: Status 404 returned error can't find the container with id d424f987ea581a2ffe9951e358b03b220b71c2080f00e4229b0fe1fb04148996 Oct 29 23:12:20.216991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:20.216959 5736 generic.go:356] "Generic (PLEG): container finished" podID="33a073fb-abcb-41fe-a6e7-72d6fedc5c8d" containerID="a598cf7863fcb9946999fddfe60c10f5b7751335ca3ca948c22f54201e0b94c1" exitCode=0 Oct 29 23:12:20.217353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:20.217037 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" event={"ID":"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d","Type":"ContainerDied","Data":"a598cf7863fcb9946999fddfe60c10f5b7751335ca3ca948c22f54201e0b94c1"} Oct 29 23:12:20.217353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:20.217087 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" event={"ID":"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d","Type":"ContainerStarted","Data":"d424f987ea581a2ffe9951e358b03b220b71c2080f00e4229b0fe1fb04148996"} Oct 29 23:12:21.457320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.457288 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:21.526541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.526456 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-xks-ca\") pod \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " Oct 29 23:12:21.526725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.526679 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-openshift-ca\") pod \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " Oct 29 23:12:21.526799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.526758 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-kube-api-access-b8tsx\" (UniqueName: \"kubernetes.io/projected/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-kube-api-access-b8tsx\") pod \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\" (UID: \"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d\") " Oct 29 23:12:21.527064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.527034 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-openshift-ca" pod "33a073fb-abcb-41fe-a6e7-72d6fedc5c8d" (UID: "33a073fb-abcb-41fe-a6e7-72d6fedc5c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:21.527360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.527337 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:21.528685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.528656 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-xks-ca" pod "33a073fb-abcb-41fe-a6e7-72d6fedc5c8d" (UID: "33a073fb-abcb-41fe-a6e7-72d6fedc5c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:12:21.529252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.529227 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-kube-api-access-b8tsx" pod "33a073fb-abcb-41fe-a6e7-72d6fedc5c8d" (UID: "33a073fb-abcb-41fe-a6e7-72d6fedc5c8d"). InnerVolumeSpecName "kube-api-access-b8tsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:21.628472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.628435 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tsx\" (UniqueName: \"kubernetes.io/projected/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-kube-api-access-b8tsx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:21.628472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:21.628467 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:22.080914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.080879 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2"] Oct 29 23:12:22.081704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.081686 5736 cpu_manager.go:414] "containerMap: removing container" podUID="33a073fb-abcb-41fe-a6e7-72d6fedc5c8d" containerName="revoke-keys" Oct 29 23:12:22.081782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.081706 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a073fb-abcb-41fe-a6e7-72d6fedc5c8d" containerName="revoke-keys" Oct 29 23:12:22.081879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.081867 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a073fb-abcb-41fe-a6e7-72d6fedc5c8d" containerName="revoke-keys" Oct 29 23:12:22.085706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.085684 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.106095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.106068 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2"] Oct 29 23:12:22.236073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.236041 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79384a5b-csgr2\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.236273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.236168 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79384a5b-csgr2\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.236353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.236342 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndz6\" (UniqueName: \"kubernetes.io/projected/3f59b8f7-354c-434c-acd6-401791fd2bec-kube-api-access-vndz6\") pod \"maas-api-revoke-sub-79384a5b-csgr2\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.236994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.236972 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" Oct 29 23:12:22.237099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.237000 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g" event={"ID":"33a073fb-abcb-41fe-a6e7-72d6fedc5c8d","Type":"ContainerDied","Data":"d424f987ea581a2ffe9951e358b03b220b71c2080f00e4229b0fe1fb04148996"} Oct 29 23:12:22.237099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.237048 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d424f987ea581a2ffe9951e358b03b220b71c2080f00e4229b0fe1fb04148996" Oct 29 23:12:22.337301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.337222 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79384a5b-csgr2\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.337463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.337312 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79384a5b-csgr2\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.337548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.337531 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndz6\" (UniqueName: \"kubernetes.io/projected/3f59b8f7-354c-434c-acd6-401791fd2bec-kube-api-access-vndz6\") pod \"maas-api-revoke-sub-79384a5b-csgr2\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.337883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.337861 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79384a5b-csgr2\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.339684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.339661 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79384a5b-csgr2\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.370172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.370143 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndz6\" (UniqueName: \"kubernetes.io/projected/3f59b8f7-354c-434c-acd6-401791fd2bec-kube-api-access-vndz6\") pod \"maas-api-revoke-sub-79384a5b-csgr2\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.407773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.407746 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:22.536730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.536692 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr"] Oct 29 23:12:22.543371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.543344 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.564468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.564442 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr"] Oct 29 23:12:22.641658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.641563 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.641658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.641640 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.641897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.641761 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a63e3-aab6-4ed5-a1f8-64ba54797461-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.641957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.641899 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbvd\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-kube-api-access-gmbvd\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.642018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.641971 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.642018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.642011 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.642129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.642118 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.642238 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.642216 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.642307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.642291 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.743079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.743040 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2"] Oct 29 23:12:22.743871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.743843 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a63e3-aab6-4ed5-a1f8-64ba54797461-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.743871 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:12:22.743863 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f59b8f7_354c_434c_acd6_401791fd2bec.slice/crio-f04b8b23736650816f7743325a2b7439a6e00c4d86076e2885c551179e67fdc8 WatchSource:0}: Error finding container f04b8b23736650816f7743325a2b7439a6e00c4d86076e2885c551179e67fdc8: Status 404 returned error can't find the container with id f04b8b23736650816f7743325a2b7439a6e00c4d86076e2885c551179e67fdc8 Oct 29 23:12:22.744086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.743941 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbvd\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-kube-api-access-gmbvd\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.744086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.743984 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.744086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.744025 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.744271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.744112 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.744271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.744216 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.744389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.744280 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.744389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.744362 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.744489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.744422 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.744747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.744714 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a63e3-aab6-4ed5-a1f8-64ba54797461-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.745046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.745023 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.745157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.745049 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.745452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.745426 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.745792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.745771 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.747786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.747762 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.748142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.748120 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.769353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.769327 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.769634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.769606 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbvd\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-kube-api-access-gmbvd\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:22.863861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:22.863816 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:23.199293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:23.199263 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr"] Oct 29 23:12:23.201181 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:12:23.201152 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277a63e3_aab6_4ed5_a1f8_64ba54797461.slice/crio-8b2996530e94ffe68aae1843f75e2b8ad7468877d690158a8396bca49fa0e76e WatchSource:0}: Error finding container 8b2996530e94ffe68aae1843f75e2b8ad7468877d690158a8396bca49fa0e76e: Status 404 returned error can't find the container with id 8b2996530e94ffe68aae1843f75e2b8ad7468877d690158a8396bca49fa0e76e Oct 29 23:12:23.204071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:23.204035 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:12:23.204155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:23.204103 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:12:23.204155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:23.204136 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:12:23.247533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:23.247502 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" event={"ID":"3f59b8f7-354c-434c-acd6-401791fd2bec","Type":"ContainerStarted","Data":"9c89226925216cca4790a0e44d2bf62c70d846bcad3fa19be278c5c9014c0529"} Oct 29 23:12:23.247712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:23.247556 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" event={"ID":"3f59b8f7-354c-434c-acd6-401791fd2bec","Type":"ContainerStarted","Data":"f04b8b23736650816f7743325a2b7439a6e00c4d86076e2885c551179e67fdc8"} Oct 29 23:12:23.249079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:23.249050 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" event={"ID":"277a63e3-aab6-4ed5-a1f8-64ba54797461","Type":"ContainerStarted","Data":"8b2996530e94ffe68aae1843f75e2b8ad7468877d690158a8396bca49fa0e76e"} Oct 29 23:12:23.292059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:23.292017 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" podStartSLOduration=1.2920049310000001 podStartE2EDuration="1.292004931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:23.291860917 +0000 UTC m=+1740.407780524" watchObservedRunningTime="2026-10-29 23:12:23.292004931 +0000 UTC m=+1740.407924536" Oct 29 23:12:24.264871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:24.264834 5736 generic.go:356] "Generic (PLEG): container finished" podID="3f59b8f7-354c-434c-acd6-401791fd2bec" containerID="9c89226925216cca4790a0e44d2bf62c70d846bcad3fa19be278c5c9014c0529" exitCode=0 Oct 29 23:12:24.265382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:24.264919 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" event={"ID":"3f59b8f7-354c-434c-acd6-401791fd2bec","Type":"ContainerDied","Data":"9c89226925216cca4790a0e44d2bf62c70d846bcad3fa19be278c5c9014c0529"} Oct 29 23:12:24.266928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:24.266905 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" event={"ID":"277a63e3-aab6-4ed5-a1f8-64ba54797461","Type":"ContainerStarted","Data":"9d5a8165f1f42601dee81bf079b5e98dc871de8d1585edfee9d5802fdfdb70ab"} Oct 29 23:12:24.355606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:24.354790 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" podStartSLOduration=2.354769445 podStartE2EDuration="2.354769445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:24.354216385 +0000 UTC m=+1741.470135993" watchObservedRunningTime="2026-10-29 23:12:24.354769445 +0000 UTC m=+1741.470689053" Oct 29 23:12:24.864887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:24.864847 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:24.866122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:24.866095 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:12:24.866232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:24.866144 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 29 23:12:25.667329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.667298 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:25.789701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.789655 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-openshift-ca\") pod \"3f59b8f7-354c-434c-acd6-401791fd2bec\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " Oct 29 23:12:25.789878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.789712 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-xks-ca\") pod \"3f59b8f7-354c-434c-acd6-401791fd2bec\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " Oct 29 23:12:25.789987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.789967 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f59b8f7-354c-434c-acd6-401791fd2bec-kube-api-access-vndz6\" (UniqueName: \"kubernetes.io/projected/3f59b8f7-354c-434c-acd6-401791fd2bec-kube-api-access-vndz6\") pod \"3f59b8f7-354c-434c-acd6-401791fd2bec\" (UID: \"3f59b8f7-354c-434c-acd6-401791fd2bec\") " Oct 29 23:12:25.790095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.790066 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-openshift-ca" pod "3f59b8f7-354c-434c-acd6-401791fd2bec" (UID: "3f59b8f7-354c-434c-acd6-401791fd2bec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:12:25.790637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.790541 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:25.792827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.792412 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-xks-ca" pod "3f59b8f7-354c-434c-acd6-401791fd2bec" (UID: "3f59b8f7-354c-434c-acd6-401791fd2bec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:12:25.792827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.792748 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f59b8f7-354c-434c-acd6-401791fd2bec-kube-api-access-vndz6" pod "3f59b8f7-354c-434c-acd6-401791fd2bec" (UID: "3f59b8f7-354c-434c-acd6-401791fd2bec"). InnerVolumeSpecName "kube-api-access-vndz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:12:25.864961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.864927 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:12:25.865162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.864979 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 29 23:12:25.891888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.891851 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vndz6\" (UniqueName: \"kubernetes.io/projected/3f59b8f7-354c-434c-acd6-401791fd2bec-kube-api-access-vndz6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:25.891888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:25.891881 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f59b8f7-354c-434c-acd6-401791fd2bec-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:12:26.460787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:26.460757 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" Oct 29 23:12:26.460787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:26.460777 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2" event={"ID":"3f59b8f7-354c-434c-acd6-401791fd2bec","Type":"ContainerDied","Data":"f04b8b23736650816f7743325a2b7439a6e00c4d86076e2885c551179e67fdc8"} Oct 29 23:12:26.461047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:26.460849 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f04b8b23736650816f7743325a2b7439a6e00c4d86076e2885c551179e67fdc8" Oct 29 23:12:26.824634 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:12:26.824537 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:12:26.865204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:26.865171 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:12:26.865373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:26.865221 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 29 23:12:27.875628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:27.875112 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:12:27.875628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:27.875177 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 29 23:12:28.865094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:28.865058 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:12:28.865289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:28.865116 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 29 23:12:29.864969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:29.864930 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:12:29.865402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:29.865000 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 29 23:12:30.870677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:30.870637 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:30.871302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:30.871281 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:30.872249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:30.872226 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:12:37.553847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.553815 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5"] Oct 29 23:12:37.554389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.554376 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3f59b8f7-354c-434c-acd6-401791fd2bec" containerName="revoke-keys" Oct 29 23:12:37.554439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.554390 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f59b8f7-354c-434c-acd6-401791fd2bec" containerName="revoke-keys" Oct 29 23:12:37.554538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.554528 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f59b8f7-354c-434c-acd6-401791fd2bec" containerName="revoke-keys" Oct 29 23:12:37.559549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.559526 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.583751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.583722 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5"] Oct 29 23:12:37.648865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.648834 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.649031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.648892 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkjc\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-kube-api-access-mtkjc\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.649031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.648945 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.649031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.649015 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414097f6-1de2-48ab-add2-979a84a13abf-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.649209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.649073 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.649209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.649122 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414097f6-1de2-48ab-add2-979a84a13abf-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.649280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.649269 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.649339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.649320 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.649398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.649387 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750186 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750242 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750284 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750369 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750425 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkjc\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-kube-api-access-mtkjc\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750459 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750485 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414097f6-1de2-48ab-add2-979a84a13abf-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750526 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750563 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414097f6-1de2-48ab-add2-979a84a13abf-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750672 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750782 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.750898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.750800 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.751066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.751041 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.751319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.751299 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414097f6-1de2-48ab-add2-979a84a13abf-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.752988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.752966 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.753151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.753131 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414097f6-1de2-48ab-add2-979a84a13abf-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.773783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.773758 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.773889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.773815 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkjc\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-kube-api-access-mtkjc\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:37.874727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:37.874661 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:38.204266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:38.204237 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5"] Oct 29 23:12:38.205766 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:12:38.205742 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414097f6_1de2_48ab_add2_979a84a13abf.slice/crio-51aa2155a9ebf75b9e432e8789dbaebc371e64086c04815c6bac1dbeddc02545 WatchSource:0}: Error finding container 51aa2155a9ebf75b9e432e8789dbaebc371e64086c04815c6bac1dbeddc02545: Status 404 returned error can't find the container with id 51aa2155a9ebf75b9e432e8789dbaebc371e64086c04815c6bac1dbeddc02545 Oct 29 23:12:38.208575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:38.208536 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:12:38.208700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:38.208631 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:12:38.208700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:38.208671 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:12:38.576047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:38.576005 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" event={"ID":"414097f6-1de2-48ab-add2-979a84a13abf","Type":"ContainerStarted","Data":"7b63381578a9dd6aad401db6b323e7b22e0dfdaaab4f20907edabc3080b0e1ba"} Oct 29 23:12:38.576524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:38.576074 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" event={"ID":"414097f6-1de2-48ab-add2-979a84a13abf","Type":"ContainerStarted","Data":"51aa2155a9ebf75b9e432e8789dbaebc371e64086c04815c6bac1dbeddc02545"} Oct 29 23:12:38.625319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:38.625271 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podStartSLOduration=1.625258132 podStartE2EDuration="1.625258132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:38.624701318 +0000 UTC m=+1755.740620925" watchObservedRunningTime="2026-10-29 23:12:38.625258132 +0000 UTC m=+1755.741177729" Oct 29 23:12:38.887656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:38.886772 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:39.882550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:39.882504 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:12:39.883108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:39.882641 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 29 23:12:40.522386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.522352 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm"] Oct 29 23:12:40.527179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.527154 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:40.565394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.564880 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-2126fb\"" Oct 29 23:12:40.565394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.565044 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-2126fb-dockercfg-6cmwf\"" Oct 29 23:12:40.570604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.567169 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm"] Oct 29 23:12:40.653604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.652142 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7cc703-61ff-4a81-b215-a5619f21af8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm\" (UID: \"ab7cc703-61ff-4a81-b215-a5619f21af8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:40.653604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.652258 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2j4p\" (UniqueName: \"kubernetes.io/projected/ab7cc703-61ff-4a81-b215-a5619f21af8a-kube-api-access-d2j4p\") pod \"payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm\" (UID: \"ab7cc703-61ff-4a81-b215-a5619f21af8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:40.771641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.770708 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7cc703-61ff-4a81-b215-a5619f21af8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm\" (UID: \"ab7cc703-61ff-4a81-b215-a5619f21af8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:40.771641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.770859 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2j4p\" (UniqueName: \"kubernetes.io/projected/ab7cc703-61ff-4a81-b215-a5619f21af8a-kube-api-access-d2j4p\") pod \"payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm\" (UID: \"ab7cc703-61ff-4a81-b215-a5619f21af8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:40.775533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.773133 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7cc703-61ff-4a81-b215-a5619f21af8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm\" (UID: \"ab7cc703-61ff-4a81-b215-a5619f21af8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:40.796161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.796134 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2j4p\" (UniqueName: \"kubernetes.io/projected/ab7cc703-61ff-4a81-b215-a5619f21af8a-kube-api-access-d2j4p\") pod \"payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm\" (UID: \"ab7cc703-61ff-4a81-b215-a5619f21af8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:40.897767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.897736 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:40.898804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.898776 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:12:40.899199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:40.899167 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 29 23:12:41.468438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:41.468399 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8"] Oct 29 23:12:41.473094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:41.473066 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-618d6c-p9sx8"] Oct 29 23:12:41.526503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:41.525244 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm"] Oct 29 23:12:41.662037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:41.661945 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" event={"ID":"ab7cc703-61ff-4a81-b215-a5619f21af8a","Type":"ContainerStarted","Data":"d41a8888bf018a10e9c7812fa60db6c030e873f0118f87748cd639ead3bb450b"} Oct 29 23:12:41.880605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:41.877150 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:12:41.880605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:41.877213 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 29 23:12:42.674675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.661702 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:42.674675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.665041 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 29 23:12:42.674675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.665370 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 23:12:42.674675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.665859 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 29 23:12:42.737615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.734566 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:42.752288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.752246 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" event={"ID":"ab7cc703-61ff-4a81-b215-a5619f21af8a","Type":"ContainerStarted","Data":"a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510"} Oct 29 23:12:42.764622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.753556 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:42.821615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.814864 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:42.821615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.818114 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" podStartSLOduration=2.818095675 podStartE2EDuration="2.818095675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:42.801902 +0000 UTC m=+1759.917821635" watchObservedRunningTime="2026-10-29 23:12:42.818095675 +0000 UTC m=+1759.934015282" Oct 29 23:12:42.890102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.890062 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:12:42.890385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.890355 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 29 23:12:42.942654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.942090 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:42.942654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.942467 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:42.994729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.994680 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 23:12:42.998693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:42.998655 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 29 23:12:43.076614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.061878 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:43.092613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.084963 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:43.271606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.230431 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 23:12:43.295603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.251388 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:43.296187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.259671 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d0207d-7a96-4e74-bdb8-b3c2de58b219" path="/var/lib/kubelet/pods/63d0207d-7a96-4e74-bdb8-b3c2de58b219/volumes" Oct 29 23:12:43.418067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.418018 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:43.425927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.420645 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 23:12:43.663665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.662677 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 29 23:12:43.663665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.663301 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 23:12:43.671602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.668654 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:43.671602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.668988 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 29 23:12:43.671602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.669240 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 23:12:43.672025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.672004 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 23:12:43.673609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.673265 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 23:12:43.674980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.673820 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:12:43.752659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.745852 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 29 23:12:43.752659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.746429 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 23:12:43.832625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.831431 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:43.833627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.833231 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:12:43.877741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.875169 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:12:43.877741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.875229 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 29 23:12:43.950986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.944363 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:43.950986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.944762 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 23:12:43.950986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.945380 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 23:12:43.950986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.945439 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:12:43.950986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.950150 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 23:12:43.964226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:43.958699 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.014818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.014771 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.036606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.015569 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 23:12:44.037786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.022438 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.038302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.022478 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.039021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.039001 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:12:44.388604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.250641 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.388604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.251019 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.388604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.251633 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:12:44.388604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.280122 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.388604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.280330 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 23:12:44.388604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.305032 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.388604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.305452 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 23:12:44.453522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.453474 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.458571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.453991 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.466597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.459467 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:12:44.482603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.459784 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.525874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.525841 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 23:12:44.656610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.656498 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:12:44.669142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.669091 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.669468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.669437 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.712786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.685425 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 29 23:12:44.741397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.741366 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 23:12:44.764604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.759726 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:44.890611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.873830 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.931179 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.934955 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.142047 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.994739 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.056883 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.056917 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.056939 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:44.994273 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.124125 5736 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:45.163606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.152173 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.299606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.299354 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 23:12:45.333600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.324738 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 23:12:45.333600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.325849 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 23:12:45.333600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.328794 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.511600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.498662 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.568719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.568686 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:12:45.586854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.583601 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:12:45.586854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.583660 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:12:45.586854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.583699 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 23:12:45.686603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.670511 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:45.787659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.786932 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" Oct 29 23:12:45.990604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.985762 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:12:45.990604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:45.985826 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 29 23:12:46.115242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.115155 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:12:46.115522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.115495 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.115706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.115688 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.172707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.172660 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 29 23:12:46.225165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.225132 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:12:46.229956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.184123 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 29 23:12:46.230420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.185782 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.233394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.233366 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.233918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.182997 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.285183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.285150 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 23:12:46.305823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.289290 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:12:46.306288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.306267 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:12:46.307506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.307469 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:46.309195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.309172 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 23:12:46.328595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.327098 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded" Oct 29 23:12:46.332598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.330429 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 29 23:12:46.390632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.390532 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:12:46.392789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.392768 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:12:46.409594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.394252 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:12:46.409594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.394312 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 23:12:46.409594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.395563 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:12:46.409594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.397805 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 23:12:46.409594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.400395 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 23:12:46.875027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.874991 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 29 23:12:46.875514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:46.875054 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 29 23:12:48.024602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:48.022106 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:48.024602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:48.024024 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:48.028600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:48.026644 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:12:50.551166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:50.551130 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g"] Oct 29 23:12:50.590087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:50.590038 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-65ffcb-tk56g"] Oct 29 23:12:51.004928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:51.004892 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a073fb-abcb-41fe-a6e7-72d6fedc5c8d" path="/var/lib/kubelet/pods/33a073fb-abcb-41fe-a6e7-72d6fedc5c8d/volumes" Oct 29 23:12:54.981895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:12:54.981865 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:12:56.824621 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:12:56.824558 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:13:04.921494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:04.921456 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq"] Oct 29 23:13:04.925798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:04.925778 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:04.946079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:04.946051 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq"] Oct 29 23:13:04.954877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:04.954853 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:13:04.955062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:04.954845 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:13:05.100505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.100472 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742bc1dd-nkwwq\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.100789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.100765 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rlnk\" (UniqueName: \"kubernetes.io/projected/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-kube-api-access-5rlnk\") pod \"maas-api-revoke-sub-742bc1dd-nkwwq\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.100915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.100896 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742bc1dd-nkwwq\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.202233 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.202148 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlnk\" (UniqueName: \"kubernetes.io/projected/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-kube-api-access-5rlnk\") pod \"maas-api-revoke-sub-742bc1dd-nkwwq\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.202381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.202258 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742bc1dd-nkwwq\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.202381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.202352 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742bc1dd-nkwwq\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.203056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.203029 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742bc1dd-nkwwq\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.204738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.204714 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742bc1dd-nkwwq\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.225977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.225951 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rlnk\" (UniqueName: \"kubernetes.io/projected/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-kube-api-access-5rlnk\") pod \"maas-api-revoke-sub-742bc1dd-nkwwq\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.241268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.241244 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:05.588215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:05.588188 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq"] Oct 29 23:13:05.590206 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:05.590181 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d00a748_d1d3_44a8_b521_eeb79bfde4ed.slice/crio-02b27b1d0e865914f3582b6ae4348b7d2a2e3fcb0aef8ebb7830a07704f7e6ee WatchSource:0}: Error finding container 02b27b1d0e865914f3582b6ae4348b7d2a2e3fcb0aef8ebb7830a07704f7e6ee: Status 404 returned error can't find the container with id 02b27b1d0e865914f3582b6ae4348b7d2a2e3fcb0aef8ebb7830a07704f7e6ee Oct 29 23:13:06.562643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:06.562571 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" event={"ID":"3d00a748-d1d3-44a8-b521-eeb79bfde4ed","Type":"ContainerStarted","Data":"4b28c8347d07a660c1e72fc4ac20b6aa58d7e9575092daaa482f4062349dfaf1"} Oct 29 23:13:06.563076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:06.562672 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" event={"ID":"3d00a748-d1d3-44a8-b521-eeb79bfde4ed","Type":"ContainerStarted","Data":"02b27b1d0e865914f3582b6ae4348b7d2a2e3fcb0aef8ebb7830a07704f7e6ee"} Oct 29 23:13:06.603894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:06.603844 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" podStartSLOduration=2.603830599 podStartE2EDuration="2.603830599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:06.602832557 +0000 UTC m=+1783.718752164" watchObservedRunningTime="2026-10-29 23:13:06.603830599 +0000 UTC m=+1783.719750456" Oct 29 23:13:07.572827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:07.572781 5736 generic.go:356] "Generic (PLEG): container finished" podID="3d00a748-d1d3-44a8-b521-eeb79bfde4ed" containerID="4b28c8347d07a660c1e72fc4ac20b6aa58d7e9575092daaa482f4062349dfaf1" exitCode=0 Oct 29 23:13:07.573251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:07.572836 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" event={"ID":"3d00a748-d1d3-44a8-b521-eeb79bfde4ed","Type":"ContainerDied","Data":"4b28c8347d07a660c1e72fc4ac20b6aa58d7e9575092daaa482f4062349dfaf1"} Oct 29 23:13:08.738989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.738966 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:08.753817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.753791 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-xks-ca\") pod \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " Oct 29 23:13:08.753961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.753949 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-kube-api-access-5rlnk\" (UniqueName: \"kubernetes.io/projected/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-kube-api-access-5rlnk\") pod \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " Oct 29 23:13:08.754011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.753978 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-openshift-ca\") pod \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\" (UID: \"3d00a748-d1d3-44a8-b521-eeb79bfde4ed\") " Oct 29 23:13:08.754418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.754390 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-openshift-ca" pod "3d00a748-d1d3-44a8-b521-eeb79bfde4ed" (UID: "3d00a748-d1d3-44a8-b521-eeb79bfde4ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:08.754561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.754541 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:13:08.755992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.755963 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-kube-api-access-5rlnk" pod "3d00a748-d1d3-44a8-b521-eeb79bfde4ed" (UID: "3d00a748-d1d3-44a8-b521-eeb79bfde4ed"). InnerVolumeSpecName "kube-api-access-5rlnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:08.755992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.755964 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-xks-ca" pod "3d00a748-d1d3-44a8-b521-eeb79bfde4ed" (UID: "3d00a748-d1d3-44a8-b521-eeb79bfde4ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:13:08.855907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.855838 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:13:08.855907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:08.855868 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rlnk\" (UniqueName: \"kubernetes.io/projected/3d00a748-d1d3-44a8-b521-eeb79bfde4ed-kube-api-access-5rlnk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:13:09.590124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:09.590076 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" event={"ID":"3d00a748-d1d3-44a8-b521-eeb79bfde4ed","Type":"ContainerDied","Data":"02b27b1d0e865914f3582b6ae4348b7d2a2e3fcb0aef8ebb7830a07704f7e6ee"} Oct 29 23:13:09.590309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:09.590127 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq" Oct 29 23:13:09.590309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:09.590160 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02b27b1d0e865914f3582b6ae4348b7d2a2e3fcb0aef8ebb7830a07704f7e6ee" Oct 29 23:13:15.781140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.781090 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq"] Oct 29 23:13:15.781998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.781973 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3d00a748-d1d3-44a8-b521-eeb79bfde4ed" containerName="revoke-keys" Oct 29 23:13:15.781998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.781998 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d00a748-d1d3-44a8-b521-eeb79bfde4ed" containerName="revoke-keys" Oct 29 23:13:15.782192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.782168 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d00a748-d1d3-44a8-b521-eeb79bfde4ed" containerName="revoke-keys" Oct 29 23:13:15.808439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.808408 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:15.819565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.819512 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq"] Oct 29 23:13:15.883520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.878611 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45b45558-qqgbq\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:15.883996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.883969 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45b45558-qqgbq\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:15.884210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.884171 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn542\" (UniqueName: \"kubernetes.io/projected/dc96235e-b95e-4849-8507-ed183ad0b87a-kube-api-access-wn542\") pod \"maas-api-revoke-sub-45b45558-qqgbq\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:15.904149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.904120 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:13:15.904279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.904177 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:13:15.993839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:15.993800 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45b45558-qqgbq\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:16.020814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:16.020445 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45b45558-qqgbq\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:16.021176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:16.021142 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn542\" (UniqueName: \"kubernetes.io/projected/dc96235e-b95e-4849-8507-ed183ad0b87a-kube-api-access-wn542\") pod \"maas-api-revoke-sub-45b45558-qqgbq\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:16.021359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:16.021334 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45b45558-qqgbq\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:16.022258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:16.022229 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45b45558-qqgbq\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:16.072658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:16.053498 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn542\" (UniqueName: \"kubernetes.io/projected/dc96235e-b95e-4849-8507-ed183ad0b87a-kube-api-access-wn542\") pod \"maas-api-revoke-sub-45b45558-qqgbq\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:16.136548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:16.136503 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:16.476937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:16.476348 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq"] Oct 29 23:13:16.484043 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:16.484014 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc96235e_b95e_4849_8507_ed183ad0b87a.slice/crio-66ceeb2226e75a9ba7296b14831a3a68037cc32971329266a1200885b2892f21 WatchSource:0}: Error finding container 66ceeb2226e75a9ba7296b14831a3a68037cc32971329266a1200885b2892f21: Status 404 returned error can't find the container with id 66ceeb2226e75a9ba7296b14831a3a68037cc32971329266a1200885b2892f21 Oct 29 23:13:16.716734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:16.716670 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" event={"ID":"dc96235e-b95e-4849-8507-ed183ad0b87a","Type":"ContainerStarted","Data":"66ceeb2226e75a9ba7296b14831a3a68037cc32971329266a1200885b2892f21"} Oct 29 23:13:17.110962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.110878 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr"] Oct 29 23:13:17.115681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.115655 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.147634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.146710 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c47a72-openshift-default-dockercfg-67hdb\"" Oct 29 23:13:17.153446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.153408 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr"] Oct 29 23:13:17.243423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.243391 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-data\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.243616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.243481 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-certs\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.243684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.243611 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2454278-070f-4050-9906-5e030d277bc3-istiod-ca-cert\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.243734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.243685 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drhjb\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-kube-api-access-drhjb\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.243734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.243719 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-envoy\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.243835 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.243805 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-socket\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.243888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.243861 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-istio-token\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.244006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.243985 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-credential-socket\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.244089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.244033 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2454278-070f-4050-9906-5e030d277bc3-istio-podinfo\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.367767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.367673 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-certs\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.368643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.368609 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-certs\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.369498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.369469 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2454278-070f-4050-9906-5e030d277bc3-istiod-ca-cert\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.369689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.368539 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2454278-070f-4050-9906-5e030d277bc3-istiod-ca-cert\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.369890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.369861 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drhjb\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-kube-api-access-drhjb\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.370418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.370393 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-envoy\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.371207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.371185 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-socket\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.371700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.371666 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-istio-token\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.372167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.372146 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-credential-socket\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.372619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.372599 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2454278-070f-4050-9906-5e030d277bc3-istio-podinfo\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.373358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.373337 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-data\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.373849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.371606 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-socket\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.373978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.372527 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-credential-socket\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.374128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.373744 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-data\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.463690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.458313 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drhjb\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-kube-api-access-drhjb\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.463690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.460438 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-istio-token\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.464936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.464883 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-envoy\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.465495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.465473 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2454278-070f-4050-9906-5e030d277bc3-istio-podinfo\") pod \"e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:17.728530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.728495 5736 generic.go:356] "Generic (PLEG): container finished" podID="dc96235e-b95e-4849-8507-ed183ad0b87a" containerID="903a756f6c825084c975f1840bd4ebefd2b8d1dc6296bd25f9befcf8dca56c67" exitCode=0 Oct 29 23:13:17.728695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.728588 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" event={"ID":"dc96235e-b95e-4849-8507-ed183ad0b87a","Type":"ContainerDied","Data":"903a756f6c825084c975f1840bd4ebefd2b8d1dc6296bd25f9befcf8dca56c67"} Oct 29 23:13:17.730767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:17.730748 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:18.055682 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:18.055637 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2454278_070f_4050_9906_5e030d277bc3.slice/crio-305d65a1c469c00064b5ff51949a54fdeeac157b9cbda7eb2e38ad1d5d49fa4b WatchSource:0}: Error finding container 305d65a1c469c00064b5ff51949a54fdeeac157b9cbda7eb2e38ad1d5d49fa4b: Status 404 returned error can't find the container with id 305d65a1c469c00064b5ff51949a54fdeeac157b9cbda7eb2e38ad1d5d49fa4b Oct 29 23:13:18.055833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:18.055678 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr"] Oct 29 23:13:18.059471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:18.059432 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:18.059594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:18.059522 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:18.059594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:18.059571 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:18.743379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:18.743285 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" event={"ID":"e2454278-070f-4050-9906-5e030d277bc3","Type":"ContainerStarted","Data":"c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453"} Oct 29 23:13:18.743379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:18.743351 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" event={"ID":"e2454278-070f-4050-9906-5e030d277bc3","Type":"ContainerStarted","Data":"305d65a1c469c00064b5ff51949a54fdeeac157b9cbda7eb2e38ad1d5d49fa4b"} Oct 29 23:13:18.807434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:18.807382 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" podStartSLOduration=1.807366356 podStartE2EDuration="1.807366356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:18.802348146 +0000 UTC m=+1795.918267754" watchObservedRunningTime="2026-10-29 23:13:18.807366356 +0000 UTC m=+1795.923285953" Oct 29 23:13:18.984056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:18.983641 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:19.042291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.042200 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc96235e-b95e-4849-8507-ed183ad0b87a-kube-api-access-wn542\" (UniqueName: \"kubernetes.io/projected/dc96235e-b95e-4849-8507-ed183ad0b87a-kube-api-access-wn542\") pod \"dc96235e-b95e-4849-8507-ed183ad0b87a\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " Oct 29 23:13:19.042291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.042273 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-xks-ca\") pod \"dc96235e-b95e-4849-8507-ed183ad0b87a\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " Oct 29 23:13:19.042529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.042457 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-openshift-ca\") pod \"dc96235e-b95e-4849-8507-ed183ad0b87a\" (UID: \"dc96235e-b95e-4849-8507-ed183ad0b87a\") " Oct 29 23:13:19.043048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.042992 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-openshift-ca" pod "dc96235e-b95e-4849-8507-ed183ad0b87a" (UID: "dc96235e-b95e-4849-8507-ed183ad0b87a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:19.043651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.043542 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:13:19.045675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.045641 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-xks-ca" pod "dc96235e-b95e-4849-8507-ed183ad0b87a" (UID: "dc96235e-b95e-4849-8507-ed183ad0b87a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:13:19.047552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.047525 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc96235e-b95e-4849-8507-ed183ad0b87a-kube-api-access-wn542" pod "dc96235e-b95e-4849-8507-ed183ad0b87a" (UID: "dc96235e-b95e-4849-8507-ed183ad0b87a"). InnerVolumeSpecName "kube-api-access-wn542". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:19.145207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.145170 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn542\" (UniqueName: \"kubernetes.io/projected/dc96235e-b95e-4849-8507-ed183ad0b87a-kube-api-access-wn542\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:13:19.145207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.145204 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc96235e-b95e-4849-8507-ed183ad0b87a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:13:19.730982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.730947 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:19.738091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.738063 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:19.752306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.752277 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" Oct 29 23:13:19.752785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.752309 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq" event={"ID":"dc96235e-b95e-4849-8507-ed183ad0b87a","Type":"ContainerDied","Data":"66ceeb2226e75a9ba7296b14831a3a68037cc32971329266a1200885b2892f21"} Oct 29 23:13:19.752785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.752353 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66ceeb2226e75a9ba7296b14831a3a68037cc32971329266a1200885b2892f21" Oct 29 23:13:19.752949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.752819 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:19.753658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:19.753639 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:13:22.764657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:22.738048 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx"] Oct 29 23:13:22.764657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:22.739009 5736 cpu_manager.go:414] "containerMap: removing container" podUID="dc96235e-b95e-4849-8507-ed183ad0b87a" containerName="revoke-keys" Oct 29 23:13:22.764657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:22.739029 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc96235e-b95e-4849-8507-ed183ad0b87a" containerName="revoke-keys" Oct 29 23:13:22.764657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:22.739169 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc96235e-b95e-4849-8507-ed183ad0b87a" containerName="revoke-keys" Oct 29 23:13:23.236603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.235607 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:23.271604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.267919 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c83b50\"" Oct 29 23:13:23.271604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.268026 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c83b50-dockercfg-jkhqf\"" Oct 29 23:13:23.390812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.390776 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx"] Oct 29 23:13:23.422432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.422390 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnsk\" (UniqueName: \"kubernetes.io/projected/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-kube-api-access-mvnsk\") pod \"payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx\" (UID: \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:23.422641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.422450 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx\" (UID: \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:23.428869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.428831 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 23:13:23.449199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.449161 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:23.482529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.482494 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 29 23:13:23.483071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.483042 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 29 23:13:23.492154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.491686 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:23.525773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.525741 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnsk\" (UniqueName: \"kubernetes.io/projected/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-kube-api-access-mvnsk\") pod \"payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx\" (UID: \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:23.525911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.525799 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx\" (UID: \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:23.527311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.527266 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx\" (UID: \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:23.528034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.527970 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:23.553612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.553569 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnsk\" (UniqueName: \"kubernetes.io/projected/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-kube-api-access-mvnsk\") pod \"payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx\" (UID: \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:23.597880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.597646 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:23.717685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.717663 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:13:23.730781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:23.730752 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:13:24.031236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:24.031133 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx"] Oct 29 23:13:24.034980 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:24.034921 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb9bb9b_227a_41ad_8026_3ff61b4cc361.slice/crio-ddb551bff68e9761fd3b7f2655ea525d97d8c97268fcbff8f9b0d23cb3a239f4 WatchSource:0}: Error finding container ddb551bff68e9761fd3b7f2655ea525d97d8c97268fcbff8f9b0d23cb3a239f4: Status 404 returned error can't find the container with id ddb551bff68e9761fd3b7f2655ea525d97d8c97268fcbff8f9b0d23cb3a239f4 Oct 29 23:13:24.055390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:24.055357 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" event={"ID":"3cb9bb9b-227a-41ad-8026-3ff61b4cc361","Type":"ContainerStarted","Data":"ddb551bff68e9761fd3b7f2655ea525d97d8c97268fcbff8f9b0d23cb3a239f4"} Oct 29 23:13:25.061646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:25.061611 5736 scope.go:122] "RemoveContainer" containerID="4012cf26354b2a99e321a01fff592f1b51f4863a5e30b99249a86bfb8235c725" Oct 29 23:13:25.065940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:25.065914 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" event={"ID":"3cb9bb9b-227a-41ad-8026-3ff61b4cc361","Type":"ContainerStarted","Data":"607353011d7df5696d9a8b811779bde9788a4004359585f16799f86f86d1707f"} Oct 29 23:13:25.066050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:25.065971 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:25.072813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:25.072793 5736 scope.go:122] "RemoveContainer" containerID="0ba6821d67b53effd418464de15c56a760b6de6ea35e2e47aa3a4bd48b8d7797" Oct 29 23:13:25.082647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:25.082628 5736 scope.go:122] "RemoveContainer" containerID="5ef9aa92a978bb7bd704fb53534d686978b02f8fa2186a6c7fa9161a99078730" Oct 29 23:13:25.110748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:25.110700 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" podStartSLOduration=3.110686733 podStartE2EDuration="3.110686733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:25.106105588 +0000 UTC m=+1802.222025224" watchObservedRunningTime="2026-10-29 23:13:25.110686733 +0000 UTC m=+1802.226606335" Oct 29 23:13:26.827068 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:13:26.827028 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:13:36.088064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.087984 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:13:36.104034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.104008 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6"] Oct 29 23:13:36.109962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.109935 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.126558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.126539 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b4b5e3-openshift-default-dockercfg-td4sl\"" Oct 29 23:13:36.131695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.131672 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6"] Oct 29 23:13:36.176562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.176522 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.176774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.176590 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.176774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.176675 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9lj\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-kube-api-access-js9lj\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.176978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.176956 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.177057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.177004 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.177128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.177066 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.177128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.177092 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.177128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.177125 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.177292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.177141 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.281420 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.281521 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.281627 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.281668 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.281733 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.281753 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.281949 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.282004 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.282109 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js9lj\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-kube-api-access-js9lj\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.283053 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.283329 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.284133 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.289237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.284409 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.304631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.303254 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.324572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.324478 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.342648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.333181 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.345603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.345538 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9lj\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-kube-api-access-js9lj\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.352143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.352114 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.425975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.425945 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:36.773098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.773056 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6"] Oct 29 23:13:36.778265 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:36.778230 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ac2122_30d3_428c_bdcb_906a6ed7e64a.slice/crio-e2cc6a4dac5f6a26ed42a9da381a4e05fe7a4165e2f6c3f001209186bd1cd426 WatchSource:0}: Error finding container e2cc6a4dac5f6a26ed42a9da381a4e05fe7a4165e2f6c3f001209186bd1cd426: Status 404 returned error can't find the container with id e2cc6a4dac5f6a26ed42a9da381a4e05fe7a4165e2f6c3f001209186bd1cd426 Oct 29 23:13:36.781925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.781884 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:36.782041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.781979 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:36.782041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:36.782022 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:37.185360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:37.185271 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" event={"ID":"b9ac2122-30d3-428c-bdcb-906a6ed7e64a","Type":"ContainerStarted","Data":"b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17"} Oct 29 23:13:37.185360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:37.185329 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" event={"ID":"b9ac2122-30d3-428c-bdcb-906a6ed7e64a","Type":"ContainerStarted","Data":"e2cc6a4dac5f6a26ed42a9da381a4e05fe7a4165e2f6c3f001209186bd1cd426"} Oct 29 23:13:37.236796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:37.236723 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" podStartSLOduration=1.236705408 podStartE2EDuration="1.236705408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:37.230507301 +0000 UTC m=+1814.346426908" watchObservedRunningTime="2026-10-29 23:13:37.236705408 +0000 UTC m=+1814.352625015" Oct 29 23:13:37.426106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:37.426072 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:37.427264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:37.427244 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:37.427329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:37.427297 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 29 23:13:38.426481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:38.426447 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:38.426941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:38.426516 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 29 23:13:39.427185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:39.427146 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:39.427706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:39.427205 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 29 23:13:40.432602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:40.426485 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:40.432602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:40.426550 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 29 23:13:41.427147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.427113 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:41.427338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.427172 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 29 23:13:41.906290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.906169 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x"] Oct 29 23:13:41.910987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.910960 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:41.944716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.944637 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x"] Oct 29 23:13:41.987874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.987841 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-certs\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:41.988154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.988126 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-data\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:41.988266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.988173 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5z8\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-kube-api-access-xn5z8\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:41.988266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.988237 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istiod-ca-cert\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:41.988386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.988287 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-credential-socket\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:41.988386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.988379 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-podinfo\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:41.988496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.988415 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-token\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:41.988554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.988509 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-envoy\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:41.988791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:41.988771 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-socket\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.008548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.008524 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-52f67b-openshift-default-dockercfg-s5n48\"" Oct 29 23:13:42.092558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.092509 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-podinfo\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.092870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.092846 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-token\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.093074 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.093057 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-envoy\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.093277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.093261 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-socket\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.093500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.093480 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-certs\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.093664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.093647 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-data\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.093797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.093782 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5z8\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-kube-api-access-xn5z8\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.093941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.093927 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istiod-ca-cert\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.094071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.094057 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-credential-socket\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.094709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.094687 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-credential-socket\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.132657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.107918 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-certs\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.132881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.110095 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-data\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.132984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.131777 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-socket\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.133075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.132568 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istiod-ca-cert\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.143623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.143569 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5z8\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-kube-api-access-xn5z8\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.144756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.144248 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-token\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.179502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.179419 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-envoy\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.180054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.180030 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-podinfo\") pod \"e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.275606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.275062 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:42.446651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.428794 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 29 23:13:42.446651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.428850 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 29 23:13:42.736108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.736076 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x"] Oct 29 23:13:42.738537 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:42.738491 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cdd1d9_d919_4a0f_95f5_59e338d9d8e4.slice/crio-8aaf21a2ec7c2ca070c8adb105b3a255d280da9f2ce81124e137f9baa9e68069 WatchSource:0}: Error finding container 8aaf21a2ec7c2ca070c8adb105b3a255d280da9f2ce81124e137f9baa9e68069: Status 404 returned error can't find the container with id 8aaf21a2ec7c2ca070c8adb105b3a255d280da9f2ce81124e137f9baa9e68069 Oct 29 23:13:42.741348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.741316 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:42.741431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.741381 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:42.741431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.741420 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:42.850659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.850630 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r"] Oct 29 23:13:42.856043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.856019 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:42.886266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.886218 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r"] Oct 29 23:13:42.931535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.931471 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hj62\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-kube-api-access-4hj62\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:42.932029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.931590 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:42.932029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.931627 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:42.932029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.931714 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:42.932029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.931897 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:42.932029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.931943 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:42.932029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.931978 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:42.932029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.932008 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:42.932453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:42.932056 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.033470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.033377 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hj62\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-kube-api-access-4hj62\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.033669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.033485 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.033669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.033519 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.033669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.033655 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.033906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.033886 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.034030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.034009 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.034229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.034210 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.034334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.034250 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.034334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.034275 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.034334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.034306 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.034505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.034353 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.034786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.034760 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.034894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.034816 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.035310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.035283 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.036655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.036630 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.037558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.037538 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.056785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.056765 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hj62\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-kube-api-access-4hj62\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.056979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.056953 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.192944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.192892 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:43.261347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.261298 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" event={"ID":"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4","Type":"ContainerStarted","Data":"5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955"} Oct 29 23:13:43.261533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.261366 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" event={"ID":"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4","Type":"ContainerStarted","Data":"8aaf21a2ec7c2ca070c8adb105b3a255d280da9f2ce81124e137f9baa9e68069"} Oct 29 23:13:43.276232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.276203 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:43.283057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.282739 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:13:43.283057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.282809 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 29 23:13:43.323609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.322311 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" podStartSLOduration=2.322289945 podStartE2EDuration="2.322289945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:43.318957871 +0000 UTC m=+1820.434877477" watchObservedRunningTime="2026-10-29 23:13:43.322289945 +0000 UTC m=+1820.438209552" Oct 29 23:13:43.463004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.462968 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:43.464363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.464334 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:43.467123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.467094 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:13:43.549063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.549033 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r"] Oct 29 23:13:43.551256 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:43.551228 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef13c0f4_a239_4b3b_9c83_c36db2353b19.slice/crio-11e2b0bbb60b7ebbb16b093048da671ca3f61e27d31207c074f249f728f39f55 WatchSource:0}: Error finding container 11e2b0bbb60b7ebbb16b093048da671ca3f61e27d31207c074f249f728f39f55: Status 404 returned error can't find the container with id 11e2b0bbb60b7ebbb16b093048da671ca3f61e27d31207c074f249f728f39f55 Oct 29 23:13:43.555155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.555114 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:43.555271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.555200 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:43.555271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:43.555243 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:44.272802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.272756 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" event={"ID":"ef13c0f4-a239-4b3b-9c83-c36db2353b19","Type":"ContainerStarted","Data":"7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a"} Oct 29 23:13:44.273339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.272820 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" event={"ID":"ef13c0f4-a239-4b3b-9c83-c36db2353b19","Type":"ContainerStarted","Data":"11e2b0bbb60b7ebbb16b093048da671ca3f61e27d31207c074f249f728f39f55"} Oct 29 23:13:44.282438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.282415 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:44.320986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.320939 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" podStartSLOduration=2.320926085 podStartE2EDuration="2.320926085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:44.317324602 +0000 UTC m=+1821.433244208" watchObservedRunningTime="2026-10-29 23:13:44.320926085 +0000 UTC m=+1821.436845691" Oct 29 23:13:44.427654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.427569 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r"] Oct 29 23:13:44.494709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.489138 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.494709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.493519 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r"] Oct 29 23:13:44.505613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.504909 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7fb940\"/\"kube-root-ca.crt\"" Oct 29 23:13:44.505613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.504974 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7fb940\"/\"openshift-service-ca.crt\"" Oct 29 23:13:44.505613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.505074 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7fb940\"/\"noten-model-7fb940-kserve-self-signed-certs\"" Oct 29 23:13:44.505613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.505524 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7fb940\"/\"default-dockercfg-jm8fn\"" Oct 29 23:13:44.579372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.579279 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-dshm\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.579552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.579413 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4411e9-7c72-4368-8f53-5b3060a40d23-tls-certs\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.579552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.579445 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-model-cache\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.579727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.579617 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-home\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.579727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.579711 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgwv\" (UniqueName: \"kubernetes.io/projected/0c4411e9-7c72-4368-8f53-5b3060a40d23-kube-api-access-vfgwv\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.579845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.579747 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-tmp-dir\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.681690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.681653 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-home\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.682042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.682016 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-home\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.682286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.682233 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgwv\" (UniqueName: \"kubernetes.io/projected/0c4411e9-7c72-4368-8f53-5b3060a40d23-kube-api-access-vfgwv\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.682492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.682471 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-tmp-dir\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.682718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.682702 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-dshm\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.682884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.682859 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-tmp-dir\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.683063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.683044 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4411e9-7c72-4368-8f53-5b3060a40d23-tls-certs\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.683184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.683170 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-model-cache\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.683725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.683703 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-model-cache\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.687234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.687213 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-dshm\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.687436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.687414 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4411e9-7c72-4368-8f53-5b3060a40d23-tls-certs\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.764603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.747326 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgwv\" (UniqueName: \"kubernetes.io/projected/0c4411e9-7c72-4368-8f53-5b3060a40d23-kube-api-access-vfgwv\") pod \"noten-model-7fb940-kserve-6ff67cd557-c6m9r\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:44.811636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:44.811544 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:45.194182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:45.194090 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:45.195888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:45.195860 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:13:45.196006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:45.195929 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 29 23:13:45.264521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:45.264495 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r"] Oct 29 23:13:45.265031 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:45.265004 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c4411e9_7c72_4368_8f53_5b3060a40d23.slice/crio-536477a3f3bc3343ee54db70a7e5e110b23907acb56297d0e3528fb1c703d1f7 WatchSource:0}: Error finding container 536477a3f3bc3343ee54db70a7e5e110b23907acb56297d0e3528fb1c703d1f7: Status 404 returned error can't find the container with id 536477a3f3bc3343ee54db70a7e5e110b23907acb56297d0e3528fb1c703d1f7 Oct 29 23:13:45.282024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:45.281988 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" event={"ID":"0c4411e9-7c72-4368-8f53-5b3060a40d23","Type":"ContainerStarted","Data":"536477a3f3bc3343ee54db70a7e5e110b23907acb56297d0e3528fb1c703d1f7"} Oct 29 23:13:45.282824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:45.282189 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:45.283220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:45.283199 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:13:46.194305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:46.194269 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:13:46.194490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:46.194318 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 29 23:13:46.291487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:46.291443 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" event={"ID":"0c4411e9-7c72-4368-8f53-5b3060a40d23","Type":"ContainerStarted","Data":"fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603"} Oct 29 23:13:46.358920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:46.358862 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" podStartSLOduration=2.358846395 podStartE2EDuration="2.358846395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:46.350984652 +0000 UTC m=+1823.466904258" watchObservedRunningTime="2026-10-29 23:13:46.358846395 +0000 UTC m=+1823.474766004" Oct 29 23:13:46.812322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:46.812231 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:47.194054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:47.193966 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:13:47.194054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:47.194026 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 29 23:13:47.971668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:47.971625 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6"] Oct 29 23:13:48.066727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.052855 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6"] Oct 29 23:13:48.066727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.053863 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:13:48.077601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.075942 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-52f67b\"" Oct 29 23:13:48.077601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.076092 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-52f67b-dockercfg-tbz9s\"" Oct 29 23:13:48.199117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.199076 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:13:48.199291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.199148 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 29 23:13:48.263657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.256695 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgm2t\" (UniqueName: \"kubernetes.io/projected/7fae0669-43fb-4822-bc69-9556cb2f0426-kube-api-access-rgm2t\") pod \"payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6\" (UID: \"7fae0669-43fb-4822-bc69-9556cb2f0426\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:13:48.263657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.256769 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fae0669-43fb-4822-bc69-9556cb2f0426-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6\" (UID: \"7fae0669-43fb-4822-bc69-9556cb2f0426\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:13:48.363176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.363127 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgm2t\" (UniqueName: \"kubernetes.io/projected/7fae0669-43fb-4822-bc69-9556cb2f0426-kube-api-access-rgm2t\") pod \"payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6\" (UID: \"7fae0669-43fb-4822-bc69-9556cb2f0426\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:13:48.363351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.363191 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fae0669-43fb-4822-bc69-9556cb2f0426-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6\" (UID: \"7fae0669-43fb-4822-bc69-9556cb2f0426\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:13:48.364334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.364307 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fae0669-43fb-4822-bc69-9556cb2f0426-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6\" (UID: \"7fae0669-43fb-4822-bc69-9556cb2f0426\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:13:48.386829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.386807 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgm2t\" (UniqueName: \"kubernetes.io/projected/7fae0669-43fb-4822-bc69-9556cb2f0426-kube-api-access-rgm2t\") pod \"payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6\" (UID: \"7fae0669-43fb-4822-bc69-9556cb2f0426\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:13:48.525245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.525155 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:13:48.862425 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:48.862395 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fae0669_43fb_4822_bc69_9556cb2f0426.slice/crio-7c3058e75bcad3ab19d447b514c28b9a40a10c4ae17c9a4e398b16b30bb70d2e WatchSource:0}: Error finding container 7c3058e75bcad3ab19d447b514c28b9a40a10c4ae17c9a4e398b16b30bb70d2e: Status 404 returned error can't find the container with id 7c3058e75bcad3ab19d447b514c28b9a40a10c4ae17c9a4e398b16b30bb70d2e Oct 29 23:13:48.863879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:48.863854 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6"] Oct 29 23:13:49.193801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:49.193725 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:13:49.194181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:49.193793 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 29 23:13:49.320742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:49.320696 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" event={"ID":"7fae0669-43fb-4822-bc69-9556cb2f0426","Type":"ContainerStarted","Data":"b09f70f23eca73fc10060b4b0e5adc4151b5fbee424cd9594c7f6c5b6164205e"} Oct 29 23:13:49.320924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:49.320761 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" event={"ID":"7fae0669-43fb-4822-bc69-9556cb2f0426","Type":"ContainerStarted","Data":"7c3058e75bcad3ab19d447b514c28b9a40a10c4ae17c9a4e398b16b30bb70d2e"} Oct 29 23:13:49.320924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:49.320812 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:13:49.381325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:49.381033 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" podStartSLOduration=2.381014868 podStartE2EDuration="2.381014868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:49.379445932 +0000 UTC m=+1826.495365539" watchObservedRunningTime="2026-10-29 23:13:49.381014868 +0000 UTC m=+1826.496934478" Oct 29 23:13:50.194365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:50.194331 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 29 23:13:50.194810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:50.194390 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 29 23:13:51.637672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:51.637637 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:51.638157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:51.638048 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:51.639174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:51.639145 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:13:54.812672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:54.812625 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:54.825362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:54.825336 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:55.389522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:55.389494 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:13:56.824814 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:13:56.824778 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:13:57.923663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:57.923617 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2"] Oct 29 23:13:57.930683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:57.930651 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:57.962926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:57.962889 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2"] Oct 29 23:13:58.054054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.054016 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.054308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.054095 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.054308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.054245 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.054399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.054309 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.054399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.054364 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.054473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.054423 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.054473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.054448 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.054682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.054658 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.054807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.054786 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24f2v\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-kube-api-access-24f2v\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.156733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.156689 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.156904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.156807 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24f2v\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-kube-api-access-24f2v\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.156995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.156918 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.156995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.156976 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.157106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.157015 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.157106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.157061 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.157106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.157091 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.157271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.157153 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.157271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.157187 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.157358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.157181 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.157358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.157308 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.157588 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.157545 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.157782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.157756 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.158149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.158125 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.159491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.159431 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.159752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.159733 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.180600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.180535 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24f2v\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-kube-api-access-24f2v\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.180600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.180533 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.291813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.291780 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:13:58.708401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.708374 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2"] Oct 29 23:13:58.709539 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:13:58.709509 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cd456c_fb72_48b1_91e5_c46d41e8dec3.slice/crio-c7bb426e850fa5bcf7b6b77c94d220208bb701e6e1b9e21def478ea07f6e48db WatchSource:0}: Error finding container c7bb426e850fa5bcf7b6b77c94d220208bb701e6e1b9e21def478ea07f6e48db: Status 404 returned error can't find the container with id c7bb426e850fa5bcf7b6b77c94d220208bb701e6e1b9e21def478ea07f6e48db Oct 29 23:13:58.712274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.712244 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:58.712340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.712312 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:58.712375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:58.712354 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:13:59.506843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:59.506791 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" event={"ID":"85cd456c-fb72-48b1-91e5-c46d41e8dec3","Type":"ContainerStarted","Data":"5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e"} Oct 29 23:13:59.507340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:59.506862 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" event={"ID":"85cd456c-fb72-48b1-91e5-c46d41e8dec3","Type":"ContainerStarted","Data":"c7bb426e850fa5bcf7b6b77c94d220208bb701e6e1b9e21def478ea07f6e48db"} Oct 29 23:13:59.584290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:13:59.584231 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" podStartSLOduration=2.584216602 podStartE2EDuration="2.584216602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:59.568422164 +0000 UTC m=+1836.684341771" watchObservedRunningTime="2026-10-29 23:13:59.584216602 +0000 UTC m=+1836.700136208" Oct 29 23:14:00.292201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:00.292160 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:14:00.293271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:00.293248 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:00.293359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:00.293300 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 29 23:14:00.356697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:00.356662 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:14:01.292984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:01.292946 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:01.293378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:01.293000 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 29 23:14:02.060438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.060401 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7"] Oct 29 23:14:02.064824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.064789 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f016f2fd-4f6x7"] Oct 29 23:14:02.068277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.068251 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg"] Oct 29 23:14:02.072849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.072831 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:02.092281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.092126 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg"] Oct 29 23:14:02.093846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.093822 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-8a8b61\"" Oct 29 23:14:02.094019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.093898 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-8a8b61-dockercfg-5574v\"" Oct 29 23:14:02.228643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.227740 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a127068b-6902-4f90-b90d-c6cc4dfcd6df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg\" (UID: \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:02.228643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.227961 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmh8\" (UniqueName: \"kubernetes.io/projected/a127068b-6902-4f90-b90d-c6cc4dfcd6df-kube-api-access-lvmh8\") pod \"payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg\" (UID: \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:02.300601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.295966 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:02.300601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.296024 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 29 23:14:02.347371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.343592 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a127068b-6902-4f90-b90d-c6cc4dfcd6df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg\" (UID: \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:02.347371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.343836 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmh8\" (UniqueName: \"kubernetes.io/projected/a127068b-6902-4f90-b90d-c6cc4dfcd6df-kube-api-access-lvmh8\") pod \"payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg\" (UID: \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:02.347371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.345275 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a127068b-6902-4f90-b90d-c6cc4dfcd6df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg\" (UID: \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:02.411075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.410261 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmh8\" (UniqueName: \"kubernetes.io/projected/a127068b-6902-4f90-b90d-c6cc4dfcd6df-kube-api-access-lvmh8\") pod \"payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg\" (UID: \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:02.688420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.688337 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:02.977486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.977450 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m"] Oct 29 23:14:02.982984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.982960 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.000706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:02.999773 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c83b50-serving-cert\"" Oct 29 23:14:03.017221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.017016 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c39fbe-50a0-44c5-abd7-f336ad491c85" path="/var/lib/kubelet/pods/12c39fbe-50a0-44c5-abd7-f336ad491c85/volumes" Oct 29 23:14:03.021490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.020701 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m"] Oct 29 23:14:03.047654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.022657 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg"] Oct 29 23:14:03.070522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.070485 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.070724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.070552 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcda232c-83c2-4a12-9577-7daabecc5f22-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.070724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.070701 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nfm\" (UniqueName: \"kubernetes.io/projected/dcda232c-83c2-4a12-9577-7daabecc5f22-kube-api-access-v7nfm\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.071026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.071001 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.172067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.172034 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nfm\" (UniqueName: \"kubernetes.io/projected/dcda232c-83c2-4a12-9577-7daabecc5f22-kube-api-access-v7nfm\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.172312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.172217 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.172312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.172280 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.172649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.172626 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcda232c-83c2-4a12-9577-7daabecc5f22-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.173279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.173251 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcda232c-83c2-4a12-9577-7daabecc5f22-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.174694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.174673 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.174807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.174764 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.195487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.195456 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nfm\" (UniqueName: \"kubernetes.io/projected/dcda232c-83c2-4a12-9577-7daabecc5f22-kube-api-access-v7nfm\") pod \"maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.301523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.301371 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:03.311432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.311144 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:03.311432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.311208 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 29 23:14:03.768436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.768162 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m"] Oct 29 23:14:03.770512 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:14:03.770466 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcda232c_83c2_4a12_9577_7daabecc5f22.slice/crio-b6976d5ec9967ab62a454dac40255b2b7904f4d02588d33ad2a0cb47155e47cd WatchSource:0}: Error finding container b6976d5ec9967ab62a454dac40255b2b7904f4d02588d33ad2a0cb47155e47cd: Status 404 returned error can't find the container with id b6976d5ec9967ab62a454dac40255b2b7904f4d02588d33ad2a0cb47155e47cd Oct 29 23:14:03.772357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.772336 5736 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:14:03.781499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.781467 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" event={"ID":"a127068b-6902-4f90-b90d-c6cc4dfcd6df","Type":"ContainerStarted","Data":"56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa"} Oct 29 23:14:03.781642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.781529 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" event={"ID":"a127068b-6902-4f90-b90d-c6cc4dfcd6df","Type":"ContainerStarted","Data":"3d6ebb7a677d545a930aa0ea8a847f70a7124d4e583302ac2dbee1fcaf5c8761"} Oct 29 23:14:03.781785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.781766 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:03.825513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:03.825430 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" podStartSLOduration=1.8254107670000002 podStartE2EDuration="1.825410767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:03.824102019 +0000 UTC m=+1840.940021625" watchObservedRunningTime="2026-10-29 23:14:03.825410767 +0000 UTC m=+1840.941330373" Oct 29 23:14:04.311342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:04.311311 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:04.311824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:04.311798 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 29 23:14:04.792522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:04.792474 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" event={"ID":"dcda232c-83c2-4a12-9577-7daabecc5f22","Type":"ContainerStarted","Data":"c0b2a07b73bbb1bb9aabdfb80eb52e8be6c9a03fa8d229e29326b65d061cb5fa"} Oct 29 23:14:04.792693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:04.792536 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" event={"ID":"dcda232c-83c2-4a12-9577-7daabecc5f22","Type":"ContainerStarted","Data":"b6976d5ec9967ab62a454dac40255b2b7904f4d02588d33ad2a0cb47155e47cd"} Oct 29 23:14:04.857310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:04.857237 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" podStartSLOduration=2.49723433 podStartE2EDuration="2.857217445s" totalImagesPullingTime="359.983115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:04.849656688 +0000 UTC m=+1841.965576294" watchObservedRunningTime="2026-10-29 23:14:04.857217445 +0000 UTC m=+1841.973137054" Oct 29 23:14:04.971601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:04.970635 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f"] Oct 29 23:14:04.991605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:04.991513 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.064225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.063876 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f"] Oct 29 23:14:05.064752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.064725 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c47a72-serving-cert\"" Oct 29 23:14:05.086692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.086501 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-metrics-certs\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.087497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.087179 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261e1e36-d96d-4e60-93cd-bab1e3e610d1-openshift-service-ca\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.087497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.087350 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.087742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.087676 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbpqq\" (UniqueName: \"kubernetes.io/projected/261e1e36-d96d-4e60-93cd-bab1e3e610d1-kube-api-access-cbpqq\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.200598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.197857 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.200598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.198202 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbpqq\" (UniqueName: \"kubernetes.io/projected/261e1e36-d96d-4e60-93cd-bab1e3e610d1-kube-api-access-cbpqq\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.200598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.198311 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-metrics-certs\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.200598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.198348 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261e1e36-d96d-4e60-93cd-bab1e3e610d1-openshift-service-ca\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.200598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.199317 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261e1e36-d96d-4e60-93cd-bab1e3e610d1-openshift-service-ca\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.200598 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:14:05.199436 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-serving-cert: secret "maas-api-e2e-explicit-c47a72-serving-cert" not found Oct 29 23:14:05.200598 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:14:05.199500 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls podName:261e1e36-d96d-4e60-93cd-bab1e3e610d1 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:05.699477887 +0000 UTC m=+1842.815397471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls") pod "maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" (UID: "261e1e36-d96d-4e60-93cd-bab1e3e610d1") : secret "maas-api-e2e-explicit-c47a72-serving-cert" not found Oct 29 23:14:05.220539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.219339 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-metrics-certs\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.227545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.227498 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbpqq\" (UniqueName: \"kubernetes.io/projected/261e1e36-d96d-4e60-93cd-bab1e3e610d1-kube-api-access-cbpqq\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.292672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.292534 5736 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:05.292672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.292608 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 29 23:14:05.706547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.706501 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:05.707041 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:14:05.706672 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-serving-cert: secret "maas-api-e2e-explicit-c47a72-serving-cert" not found Oct 29 23:14:05.707041 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:14:05.706734 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls podName:261e1e36-d96d-4e60-93cd-bab1e3e610d1 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:06.706719154 +0000 UTC m=+1843.822638737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls") pod "maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" (UID: "261e1e36-d96d-4e60-93cd-bab1e3e610d1") : secret "maas-api-e2e-explicit-c47a72-serving-cert" not found Oct 29 23:14:05.802937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:05.802896 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:06.748311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:06.748276 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:06.750859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:06.750833 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls\") pod \"maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:06.760833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:06.760807 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:14:06.761377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:06.761350 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:14:06.762000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:06.761981 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:14:06.882743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:06.882709 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:07.212068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:07.212036 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f"] Oct 29 23:14:07.212791 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:14:07.212763 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261e1e36_d96d_4e60_93cd_bab1e3e610d1.slice/crio-8caa9152e8dc5893db4f7f94dd68ea58db5156e18f72a3dba462530f0087115c WatchSource:0}: Error finding container 8caa9152e8dc5893db4f7f94dd68ea58db5156e18f72a3dba462530f0087115c: Status 404 returned error can't find the container with id 8caa9152e8dc5893db4f7f94dd68ea58db5156e18f72a3dba462530f0087115c Oct 29 23:14:07.825280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:07.825180 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" event={"ID":"261e1e36-d96d-4e60-93cd-bab1e3e610d1","Type":"ContainerStarted","Data":"9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2"} Oct 29 23:14:07.825280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:07.825248 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" event={"ID":"261e1e36-d96d-4e60-93cd-bab1e3e610d1","Type":"ContainerStarted","Data":"8caa9152e8dc5893db4f7f94dd68ea58db5156e18f72a3dba462530f0087115c"} Oct 29 23:14:07.825280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:07.825276 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:07.870837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:07.870778 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" podStartSLOduration=3.530069129 podStartE2EDuration="3.870764373s" totalImagesPullingTime="340.695244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:07.868910171 +0000 UTC m=+1844.984829777" watchObservedRunningTime="2026-10-29 23:14:07.870764373 +0000 UTC m=+1844.986683979" Oct 29 23:14:11.816209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:11.816178 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:14:13.332464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:13.332430 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r"] Oct 29 23:14:13.842650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:13.842618 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:14:14.795885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:14.795856 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:14:21.245097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.245058 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r"] Oct 29 23:14:21.249516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.249490 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.272857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.272835 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r"] Oct 29 23:14:21.345219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.345188 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.345377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.345300 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.345377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.345354 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxfd\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-kube-api-access-sfxfd\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.345471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.345425 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.345514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.345482 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138b5b65-38aa-4c67-916f-1047e5898300-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.345556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.345544 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138b5b65-38aa-4c67-916f-1047e5898300-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.345666 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.345653 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.345712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.345693 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.345750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.345712 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.447542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.447504 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.447718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.447616 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138b5b65-38aa-4c67-916f-1047e5898300-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.447718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.447652 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138b5b65-38aa-4c67-916f-1047e5898300-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.447812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.447756 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.447866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.447811 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.447866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.447840 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.447978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.447875 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.447978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.447968 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.448077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.447996 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxfd\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-kube-api-access-sfxfd\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.448305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.448279 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.448499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.448359 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.448499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.448391 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.448714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.448477 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.448714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.448551 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138b5b65-38aa-4c67-916f-1047e5898300-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.450107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.450083 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.450363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.450341 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138b5b65-38aa-4c67-916f-1047e5898300-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.471080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.471054 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.471173 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.471143 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxfd\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-kube-api-access-sfxfd\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.564678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.564564 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:21.893152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.893108 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r"] Oct 29 23:14:21.898950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.898915 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:14:21.899057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.898998 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:14:21.899057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.899049 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:14:21.950793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:21.950752 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" event={"ID":"138b5b65-38aa-4c67-916f-1047e5898300","Type":"ContainerStarted","Data":"47831ffbce52db2fe92c54eaff38a5f39fba5868e4022824a27c5c98a2c7d5cf"} Oct 29 23:14:23.080603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:23.078567 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" event={"ID":"138b5b65-38aa-4c67-916f-1047e5898300","Type":"ContainerStarted","Data":"ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60"} Oct 29 23:14:23.144318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:23.144247 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" podStartSLOduration=2.144225917 podStartE2EDuration="2.144225917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:23.130605426 +0000 UTC m=+1860.246525035" watchObservedRunningTime="2026-10-29 23:14:23.144225917 +0000 UTC m=+1860.260145524" Oct 29 23:14:23.566158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:23.566123 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:23.569614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:23.568916 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 29 23:14:23.569614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:23.568995 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" podUID="138b5b65-38aa-4c67-916f-1047e5898300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 29 23:14:24.565812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:24.565780 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 29 23:14:24.566232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:24.565846 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" podUID="138b5b65-38aa-4c67-916f-1047e5898300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 29 23:14:25.124247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:25.124208 5736 scope.go:122] "RemoveContainer" containerID="378e6cfc2d8150ebbbe0eb3fba1edf1195c92829ea751d45286f6786113bee88" Oct 29 23:14:25.161899 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:25.161870 5736 scope.go:122] "RemoveContainer" containerID="6220bcf8593d148d3b180518dc3b91367df21e902e29919a648e3f7f2f4f96a5" Oct 29 23:14:25.173164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:25.173142 5736 scope.go:122] "RemoveContainer" containerID="04af90b93b20720b203418957724b4cb572974f62df6d684f1d540aecc5179ae" Oct 29 23:14:25.183511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:25.183486 5736 scope.go:122] "RemoveContainer" containerID="1c8a74b3160ea11a37ee78ffe2bf42a6de704b8c2563ba50f69d26f2bc5ef9fa" Oct 29 23:14:25.192980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:25.192959 5736 scope.go:122] "RemoveContainer" containerID="42ffb3cdd15c429513503ba88cccb9a154beaef60c3b6094c0c2593f85d50c1c" Oct 29 23:14:25.565225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:25.565191 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 29 23:14:25.565422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:25.565247 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" podUID="138b5b65-38aa-4c67-916f-1047e5898300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 29 23:14:26.565790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:26.565752 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 29 23:14:26.566212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:26.565808 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" podUID="138b5b65-38aa-4c67-916f-1047e5898300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 29 23:14:26.825793 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:14:26.825711 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:14:27.565488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:27.565452 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 29 23:14:27.565673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:27.565507 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" podUID="138b5b65-38aa-4c67-916f-1047e5898300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 29 23:14:28.372431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.372388 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" podUID="0c4411e9-7c72-4368-8f53-5b3060a40d23" containerName="main" containerID="cri-o://fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603" gracePeriod=45 Oct 29 23:14:28.377402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.377368 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" podUID="0c4411e9-7c72-4368-8f53-5b3060a40d23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.214:8000/ready\": dial tcp 10.134.0.214:8000: connect: connection refused" Oct 29 23:14:28.616375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.616344 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:28.617268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.617242 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:28.618209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.618186 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:14:28.661836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.661811 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:14:28.714045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.714015 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-home\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-home\") pod \"0c4411e9-7c72-4368-8f53-5b3060a40d23\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " Oct 29 23:14:28.714178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.714146 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-dshm\") pod \"0c4411e9-7c72-4368-8f53-5b3060a40d23\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " Oct 29 23:14:28.714233 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.714178 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4411e9-7c72-4368-8f53-5b3060a40d23-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4411e9-7c72-4368-8f53-5b3060a40d23-tls-certs\") pod \"0c4411e9-7c72-4368-8f53-5b3060a40d23\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " Oct 29 23:14:28.714296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.714269 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-model-cache\") pod \"0c4411e9-7c72-4368-8f53-5b3060a40d23\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " Oct 29 23:14:28.714365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.714316 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-tmp-dir\") pod \"0c4411e9-7c72-4368-8f53-5b3060a40d23\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " Oct 29 23:14:28.714365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.714325 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-home" pod "0c4411e9-7c72-4368-8f53-5b3060a40d23" (UID: "0c4411e9-7c72-4368-8f53-5b3060a40d23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:28.714365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.714355 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4411e9-7c72-4368-8f53-5b3060a40d23-kube-api-access-vfgwv\" (UniqueName: \"kubernetes.io/projected/0c4411e9-7c72-4368-8f53-5b3060a40d23-kube-api-access-vfgwv\") pod \"0c4411e9-7c72-4368-8f53-5b3060a40d23\" (UID: \"0c4411e9-7c72-4368-8f53-5b3060a40d23\") " Oct 29 23:14:28.714570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.714539 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-model-cache" pod "0c4411e9-7c72-4368-8f53-5b3060a40d23" (UID: "0c4411e9-7c72-4368-8f53-5b3060a40d23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:28.714786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.714744 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-tmp-dir" pod "0c4411e9-7c72-4368-8f53-5b3060a40d23" (UID: "0c4411e9-7c72-4368-8f53-5b3060a40d23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:28.715912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.715437 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:14:28.715912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.715463 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:14:28.715912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.715475 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:14:28.718007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.717969 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4411e9-7c72-4368-8f53-5b3060a40d23-tls-certs" pod "0c4411e9-7c72-4368-8f53-5b3060a40d23" (UID: "0c4411e9-7c72-4368-8f53-5b3060a40d23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:28.718516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.718308 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4411e9-7c72-4368-8f53-5b3060a40d23-kube-api-access-vfgwv" pod "0c4411e9-7c72-4368-8f53-5b3060a40d23" (UID: "0c4411e9-7c72-4368-8f53-5b3060a40d23"). InnerVolumeSpecName "kube-api-access-vfgwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:28.724996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.724966 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-dshm" pod "0c4411e9-7c72-4368-8f53-5b3060a40d23" (UID: "0c4411e9-7c72-4368-8f53-5b3060a40d23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:28.817711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.817676 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4411e9-7c72-4368-8f53-5b3060a40d23-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:14:28.817871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.817715 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgwv\" (UniqueName: \"kubernetes.io/projected/0c4411e9-7c72-4368-8f53-5b3060a40d23-kube-api-access-vfgwv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:14:28.817871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:28.817730 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4411e9-7c72-4368-8f53-5b3060a40d23-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:14:29.116393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:29.116352 5736 generic.go:356] "Generic (PLEG): container finished" podID="0c4411e9-7c72-4368-8f53-5b3060a40d23" containerID="fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603" exitCode=0 Oct 29 23:14:29.116642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:29.116438 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" Oct 29 23:14:29.116642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:29.116464 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" event={"ID":"0c4411e9-7c72-4368-8f53-5b3060a40d23","Type":"ContainerDied","Data":"fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603"} Oct 29 23:14:29.116642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:29.116592 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r" event={"ID":"0c4411e9-7c72-4368-8f53-5b3060a40d23","Type":"ContainerDied","Data":"536477a3f3bc3343ee54db70a7e5e110b23907acb56297d0e3528fb1c703d1f7"} Oct 29 23:14:29.116642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:29.116597 5736 scope.go:122] "RemoveContainer" containerID="fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603" Oct 29 23:14:29.131914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:29.131888 5736 scope.go:122] "RemoveContainer" containerID="fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603" Oct 29 23:14:29.132280 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:14:29.132254 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603\": container with ID starting with fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603 not found: ID does not exist" containerID="fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603" Oct 29 23:14:29.132369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:29.132288 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603"} err="failed to get container status \"fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603\": rpc error: code = NotFound desc = could not find container \"fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603\": container with ID starting with fef91d9a7898bfa4ff2663d52dbd23cf34ae514c3049e32bc359b54c15031603 not found: ID does not exist" Oct 29 23:14:29.171101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:29.171066 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r"] Oct 29 23:14:29.175254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:29.175227 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-7fb940/noten-model-7fb940-kserve-6ff67cd557-c6m9r"] Oct 29 23:14:31.066931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.066899 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4411e9-7c72-4368-8f53-5b3060a40d23" path="/var/lib/kubelet/pods/0c4411e9-7c72-4368-8f53-5b3060a40d23/volumes" Oct 29 23:14:31.418675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.418570 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-2d57f6"] Oct 29 23:14:31.419531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.419509 5736 cpu_manager.go:414] "containerMap: removing container" podUID="0c4411e9-7c72-4368-8f53-5b3060a40d23" containerName="main" Oct 29 23:14:31.419531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.419527 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4411e9-7c72-4368-8f53-5b3060a40d23" containerName="main" Oct 29 23:14:31.419775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.419757 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4411e9-7c72-4368-8f53-5b3060a40d23" containerName="main" Oct 29 23:14:31.448607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.434844 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-2d57f6" Oct 29 23:14:31.449651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.449617 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-2d57f6"] Oct 29 23:14:31.452680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.450895 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xdwj4\"" Oct 29 23:14:31.506725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.506691 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkd7k\" (UniqueName: \"kubernetes.io/projected/1d893afd-4888-4166-b6c7-65227e9721b1-kube-api-access-vkd7k\") pod \"test-curl-7899-2d57f6\" (UID: \"1d893afd-4888-4166-b6c7-65227e9721b1\") " pod="opendatahub/test-curl-7899-2d57f6" Oct 29 23:14:31.608538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.608507 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkd7k\" (UniqueName: \"kubernetes.io/projected/1d893afd-4888-4166-b6c7-65227e9721b1-kube-api-access-vkd7k\") pod \"test-curl-7899-2d57f6\" (UID: \"1d893afd-4888-4166-b6c7-65227e9721b1\") " pod="opendatahub/test-curl-7899-2d57f6" Oct 29 23:14:31.656169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.656132 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkd7k\" (UniqueName: \"kubernetes.io/projected/1d893afd-4888-4166-b6c7-65227e9721b1-kube-api-access-vkd7k\") pod \"test-curl-7899-2d57f6\" (UID: \"1d893afd-4888-4166-b6c7-65227e9721b1\") " pod="opendatahub/test-curl-7899-2d57f6" Oct 29 23:14:31.795840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:31.795805 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-2d57f6" Oct 29 23:14:32.128236 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:14:32.128201 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d893afd_4888_4166_b6c7_65227e9721b1.slice/crio-ecdf9883aeaf2706e2afb384aa96443b078551837b02de4e4f809c74d996ca1f WatchSource:0}: Error finding container ecdf9883aeaf2706e2afb384aa96443b078551837b02de4e4f809c74d996ca1f: Status 404 returned error can't find the container with id ecdf9883aeaf2706e2afb384aa96443b078551837b02de4e4f809c74d996ca1f Oct 29 23:14:32.128700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:32.128287 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-2d57f6"] Oct 29 23:14:32.172228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:32.172193 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-2d57f6" event={"ID":"1d893afd-4888-4166-b6c7-65227e9721b1","Type":"ContainerStarted","Data":"ecdf9883aeaf2706e2afb384aa96443b078551837b02de4e4f809c74d996ca1f"} Oct 29 23:14:33.194641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:33.194600 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-2d57f6" event={"ID":"1d893afd-4888-4166-b6c7-65227e9721b1","Type":"ContainerStarted","Data":"6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019"} Oct 29 23:14:33.239606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:33.236857 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7899-2d57f6" podStartSLOduration=2.236839387 podStartE2EDuration="2.236839387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:33.234885599 +0000 UTC m=+1870.350805205" watchObservedRunningTime="2026-10-29 23:14:33.236839387 +0000 UTC m=+1870.352758995" Oct 29 23:14:34.328425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:34.328382 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-2d57f6"] Oct 29 23:14:34.360845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:34.360808 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-2d57f6"] Oct 29 23:14:35.212444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.212403 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7899-2d57f6" podUID="1d893afd-4888-4166-b6c7-65227e9721b1" containerName="test-curl-7899-2d57f6" containerID="cri-o://6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019" gracePeriod=2 Oct 29 23:14:35.374225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.374189 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg"] Oct 29 23:14:35.375794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.375767 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1d893afd-4888-4166-b6c7-65227e9721b1" containerName="test-curl-7899-2d57f6" Oct 29 23:14:35.375953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.375938 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d893afd-4888-4166-b6c7-65227e9721b1" containerName="test-curl-7899-2d57f6" Oct 29 23:14:35.376227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.376213 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d893afd-4888-4166-b6c7-65227e9721b1" containerName="test-curl-7899-2d57f6" Oct 29 23:14:35.381255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.381237 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.415434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.415400 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-2591f6-openshift-default-dockercfg-5rxmx\"" Oct 29 23:14:35.430634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.426265 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg"] Oct 29 23:14:35.758695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.754358 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfsxj\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-kube-api-access-sfsxj\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.758695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.754440 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-data\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.758695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.754571 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-token\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.758695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.754676 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.758695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.754715 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istiod-ca-cert\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.758695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.754878 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-envoy\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.793744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.783847 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-credential-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.793744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.783929 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-certs\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.793744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.784009 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-podinfo\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.927002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.926959 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-envoy\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.927207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.927157 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-credential-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.927273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.927208 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-certs\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.927273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.927260 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-podinfo\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.927424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.927405 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfsxj\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-kube-api-access-sfsxj\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.927482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.927450 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-data\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.927600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.927568 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-token\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.927676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.927663 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.927731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.927692 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istiod-ca-cert\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.928186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.928160 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-certs\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.929519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.929491 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-credential-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.929692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.929527 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-data\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.930055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.930029 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.930445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.930424 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istiod-ca-cert\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.940607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.937330 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-envoy\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.977018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.962723 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-podinfo\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.977018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.973140 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-token\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:35.977018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:35.975365 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfsxj\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-kube-api-access-sfsxj\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:36.236110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:36.235828 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:36.616792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:36.616762 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg"] Oct 29 23:14:36.618646 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:14:36.618574 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb002e9fa_a1b2_4a58_9468_0a6de1a35017.slice/crio-cf40de5621727c3573d32fb8cb776c4226c09946cbae907fcf1a613280de7c4f WatchSource:0}: Error finding container cf40de5621727c3573d32fb8cb776c4226c09946cbae907fcf1a613280de7c4f: Status 404 returned error can't find the container with id cf40de5621727c3573d32fb8cb776c4226c09946cbae907fcf1a613280de7c4f Oct 29 23:14:36.621799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:36.621766 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:14:36.621925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:36.621831 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:14:36.621925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:36.621868 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:14:37.258806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:37.258772 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" event={"ID":"b002e9fa-a1b2-4a58-9468-0a6de1a35017","Type":"ContainerStarted","Data":"27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61"} Oct 29 23:14:37.260058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:37.260030 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" event={"ID":"b002e9fa-a1b2-4a58-9468-0a6de1a35017","Type":"ContainerStarted","Data":"cf40de5621727c3573d32fb8cb776c4226c09946cbae907fcf1a613280de7c4f"} Oct 29 23:14:37.328479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:37.328421 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" podStartSLOduration=2.328401372 podStartE2EDuration="2.328401372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:37.327136949 +0000 UTC m=+1874.443056556" watchObservedRunningTime="2026-10-29 23:14:37.328401372 +0000 UTC m=+1874.444320979" Oct 29 23:14:37.644189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:37.644157 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-2d57f6" Oct 29 23:14:37.735827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:37.735786 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d893afd-4888-4166-b6c7-65227e9721b1-kube-api-access-vkd7k\" (UniqueName: \"kubernetes.io/projected/1d893afd-4888-4166-b6c7-65227e9721b1-kube-api-access-vkd7k\") pod \"1d893afd-4888-4166-b6c7-65227e9721b1\" (UID: \"1d893afd-4888-4166-b6c7-65227e9721b1\") " Oct 29 23:14:37.738906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:37.738878 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d893afd-4888-4166-b6c7-65227e9721b1-kube-api-access-vkd7k" pod "1d893afd-4888-4166-b6c7-65227e9721b1" (UID: "1d893afd-4888-4166-b6c7-65227e9721b1"). InnerVolumeSpecName "kube-api-access-vkd7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:37.838235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:37.838155 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkd7k\" (UniqueName: \"kubernetes.io/projected/1d893afd-4888-4166-b6c7-65227e9721b1-kube-api-access-vkd7k\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:14:38.238278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:38.238236 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:38.244398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:38.244373 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:38.270991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:38.270961 5736 generic.go:356] "Generic (PLEG): container finished" podID="1d893afd-4888-4166-b6c7-65227e9721b1" containerID="6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019" exitCode=137 Oct 29 23:14:38.271167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:38.271010 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-2d57f6" Oct 29 23:14:38.271167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:38.271063 5736 scope.go:122] "RemoveContainer" containerID="6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019" Oct 29 23:14:38.271792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:38.271406 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:38.272667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:38.272480 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:14:38.282666 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:38.282651 5736 scope.go:122] "RemoveContainer" containerID="6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019" Oct 29 23:14:38.282896 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:14:38.282875 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019\": container with ID starting with 6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019 not found: ID does not exist" containerID="6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019" Oct 29 23:14:38.282960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:38.282908 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019"} err="failed to get container status \"6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019\": rpc error: code = NotFound desc = could not find container \"6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019\": container with ID starting with 6b16ee0d6386f3952f2c6c7150ecadf3f19fa042fc5ecec4f8532b9adbb32019 not found: ID does not exist" Oct 29 23:14:38.362201 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:14:38.362154 5736 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7899-2d57f6\" is forbidden: User \"system:node:ip-10-0-143-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-172.ec2.internal' and this object" podUID="1d893afd-4888-4166-b6c7-65227e9721b1" pod="opendatahub/test-curl-7899-2d57f6" Oct 29 23:14:39.007979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:39.007918 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d893afd-4888-4166-b6c7-65227e9721b1" path="/var/lib/kubelet/pods/1d893afd-4888-4166-b6c7-65227e9721b1/volumes" Oct 29 23:14:40.219095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.219061 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk"] Oct 29 23:14:40.224720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.224696 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.247500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.247469 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk"] Oct 29 23:14:40.251893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.251859 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-96cd49\"/\"praxis-route-96cd49-kserve-self-signed-certs\"" Oct 29 23:14:40.252015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.251989 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-96cd49\"/\"default-dockercfg-g6799\"" Oct 29 23:14:40.254553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.254531 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-96cd49\"/\"openshift-service-ca.crt\"" Oct 29 23:14:40.254706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.254611 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-96cd49\"/\"kube-root-ca.crt\"" Oct 29 23:14:40.287972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.287945 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-model-cache\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.288108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.288002 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tmp-dir\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.288215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.288192 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-home\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.288399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.288378 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfh7\" (UniqueName: \"kubernetes.io/projected/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-kube-api-access-plfh7\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.288449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.288420 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-dshm\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.288526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.288502 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tls-certs\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.390187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.390148 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-model-cache\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.390389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.390229 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tmp-dir\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.390389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.390361 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-home\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.390570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.390545 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfh7\" (UniqueName: \"kubernetes.io/projected/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-kube-api-access-plfh7\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.390705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.390603 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-dshm\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.390705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.390633 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tls-certs\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.390842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.390773 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-home\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.390842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.390799 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tmp-dir\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.391366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.391338 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-model-cache\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.393679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.393653 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-dshm\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.408944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.396108 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tls-certs\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.423241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.423210 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfh7\" (UniqueName: \"kubernetes.io/projected/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-kube-api-access-plfh7\") pod \"praxis-route-96cd49-kserve-fb645f9d7-vzzwk\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:40.550058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:40.548920 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:41.013451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:41.013422 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk"] Oct 29 23:14:41.015143 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:14:41.015101 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6234f8b4_47e6_47e3_9f0b_dba6cccc5434.slice/crio-f9e7c16e28baf7fe4127dbb453a2638af82fc67effc6b90209a344d15ece2f12 WatchSource:0}: Error finding container f9e7c16e28baf7fe4127dbb453a2638af82fc67effc6b90209a344d15ece2f12: Status 404 returned error can't find the container with id f9e7c16e28baf7fe4127dbb453a2638af82fc67effc6b90209a344d15ece2f12 Oct 29 23:14:41.347455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:41.347367 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" event={"ID":"6234f8b4-47e6-47e3-9f0b-dba6cccc5434","Type":"ContainerStarted","Data":"3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0"} Oct 29 23:14:41.347455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:41.347410 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" event={"ID":"6234f8b4-47e6-47e3-9f0b-dba6cccc5434","Type":"ContainerStarted","Data":"f9e7c16e28baf7fe4127dbb453a2638af82fc67effc6b90209a344d15ece2f12"} Oct 29 23:14:41.395093 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:41.394996 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" podStartSLOduration=1.394981427 podStartE2EDuration="1.394981427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:41.389851294 +0000 UTC m=+1878.505770899" watchObservedRunningTime="2026-10-29 23:14:41.394981427 +0000 UTC m=+1878.510901032" Oct 29 23:14:41.552113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:41.549361 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:42.449116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:42.449073 5736 generic.go:356] "Generic (PLEG): container finished" podID="e2454278-070f-4050-9906-5e030d277bc3" containerID="c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453" exitCode=0 Oct 29 23:14:42.475605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:42.452409 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" event={"ID":"e2454278-070f-4050-9906-5e030d277bc3","Type":"ContainerDied","Data":"c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453"} Oct 29 23:14:42.475605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:42.452950 5736 scope.go:122] "RemoveContainer" containerID="c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453" Oct 29 23:14:42.593602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:42.578773 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:14:42.593602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:42.578935 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:14:42.593602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:42.579028 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:14:43.464458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:43.464417 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" event={"ID":"e2454278-070f-4050-9906-5e030d277bc3","Type":"ContainerStarted","Data":"663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1"} Oct 29 23:14:43.745658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:43.740040 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:14:43.745658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:43.740182 5736 patch_prober.go:29] interesting pod/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 29 23:14:43.745658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:43.740238 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 29 23:14:44.750599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:44.749893 5736 patch_prober.go:29] interesting pod/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 29 23:14:44.750599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:44.749964 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 29 23:14:45.742838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:45.742803 5736 patch_prober.go:29] interesting pod/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 29 23:14:45.743130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:45.743104 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 29 23:14:46.745612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:46.744212 5736 patch_prober.go:29] interesting pod/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 29 23:14:46.745612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:46.744285 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 29 23:14:47.739606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:47.735897 5736 patch_prober.go:29] interesting pod/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 29 23:14:47.739606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:47.735958 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 29 23:14:48.737608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:48.734932 5736 patch_prober.go:29] interesting pod/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 29 23:14:48.737608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:48.735011 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 29 23:14:49.682146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.682109 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm"] Oct 29 23:14:49.687893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.687863 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.705610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.705567 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm"] Oct 29 23:14:49.711888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.711863 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b73a75\"/\"openshift-service-ca.crt\"" Oct 29 23:14:49.712009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.711866 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b73a75\"/\"kube-root-ca.crt\"" Oct 29 23:14:49.715737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.715711 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-dshm\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.715867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.715755 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tls-certs\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.715867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.715859 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tmp-dir\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.716000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.715972 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-home\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.716094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.716076 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4wv\" (UniqueName: \"kubernetes.io/projected/4eec5768-62f2-4d97-b6dc-4f9971e8e919-kube-api-access-vg4wv\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.716192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.716171 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-model-cache\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.736890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.736863 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:14:49.737571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.737512 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:14:49.738508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.738487 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:14:49.782333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.782307 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b73a75\"/\"default-dockercfg-c4js8\"" Oct 29 23:14:49.782461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.782387 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b73a75\"/\"auth-test-model-b73a75-kserve-self-signed-certs\"" Oct 29 23:14:49.817562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.817531 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-dshm\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.817695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.817614 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tls-certs\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.817776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.817703 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tmp-dir\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.817839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.817828 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-home\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.818118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.818074 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4wv\" (UniqueName: \"kubernetes.io/projected/4eec5768-62f2-4d97-b6dc-4f9971e8e919-kube-api-access-vg4wv\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.818118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.818110 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tmp-dir\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.818308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.818179 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-model-cache\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.818308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.818188 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-home\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.818509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.818473 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-model-cache\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.820387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.820366 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-dshm\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.820628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.820574 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tls-certs\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:49.880178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:49.880137 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4wv\" (UniqueName: \"kubernetes.io/projected/4eec5768-62f2-4d97-b6dc-4f9971e8e919-kube-api-access-vg4wv\") pod \"auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:50.018604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:50.017948 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:50.492317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:50.492294 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm"] Oct 29 23:14:50.493944 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:14:50.493914 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eec5768_62f2_4d97_b6dc_4f9971e8e919.slice/crio-c3e06fb324fee083c27eccff0ce2eff4d8bdf172906c9097812b6e568a493657 WatchSource:0}: Error finding container c3e06fb324fee083c27eccff0ce2eff4d8bdf172906c9097812b6e568a493657: Status 404 returned error can't find the container with id c3e06fb324fee083c27eccff0ce2eff4d8bdf172906c9097812b6e568a493657 Oct 29 23:14:50.529521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:50.529487 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" event={"ID":"4eec5768-62f2-4d97-b6dc-4f9971e8e919","Type":"ContainerStarted","Data":"c3e06fb324fee083c27eccff0ce2eff4d8bdf172906c9097812b6e568a493657"} Oct 29 23:14:50.550141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:50.550105 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:50.562923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:50.562904 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:51.546951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:51.546905 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" event={"ID":"4eec5768-62f2-4d97-b6dc-4f9971e8e919","Type":"ContainerStarted","Data":"99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2"} Oct 29 23:14:51.559316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:51.559286 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:14:51.589373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:51.589325 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" podStartSLOduration=2.58931308 podStartE2EDuration="2.58931308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:51.587163549 +0000 UTC m=+1888.703083170" watchObservedRunningTime="2026-10-29 23:14:51.58931308 +0000 UTC m=+1888.705232715" Oct 29 23:14:52.019441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:14:52.019408 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:14:56.825333 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:14:56.825295 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:15:00.019162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.019127 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:15:00.032924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.032892 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:15:00.156544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.156511 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg"] Oct 29 23:15:00.161462 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.161438 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:00.161647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.161625 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns"] Oct 29 23:15:00.166321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.166293 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp"] Oct 29 23:15:00.166455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.166386 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:00.174199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.174177 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2"] Oct 29 23:15:00.174302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.174267 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:00.178819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.178795 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:00.180399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.180377 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:15:00.183912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.183887 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg"] Oct 29 23:15:00.189934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.189891 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp"] Oct 29 23:15:00.198606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.197837 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn"] Oct 29 23:15:00.207597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.207561 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:00.209282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.209257 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q"] Oct 29 23:15:00.214797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.214775 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:00.214998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.214977 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns"] Oct 29 23:15:00.225427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.223920 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2"] Oct 29 23:15:00.247970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.239098 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k"] Oct 29 23:15:00.252511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.251988 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q"] Oct 29 23:15:00.252511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.252035 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn"] Oct 29 23:15:00.252511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.252130 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:00.255490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.255468 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6"] Oct 29 23:15:00.262401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.261038 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:00.265433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.265416 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k"] Oct 29 23:15:00.272683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.271070 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj"] Oct 29 23:15:00.276467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.276339 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:00.287023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.286999 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6"] Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.287416 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj"] Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.287517 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b89fbf3-5d28-4308-80fc-d7d51192e711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q\" (UID: \"8b89fbf3-5d28-4308-80fc-d7d51192e711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.287605 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d879n\" (UniqueName: \"kubernetes.io/projected/108ecc68-18ab-40e6-a004-6664e74fb746-kube-api-access-d879n\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn\" (UID: \"108ecc68-18ab-40e6-a004-6664e74fb746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.287708 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dab4a21-963e-4dde-9bb1-835f436ad9a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2\" (UID: \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.287761 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a141e7-617e-4172-9c0c-d19df1046266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp\" (UID: \"83a141e7-617e-4172-9c0c-d19df1046266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.287880 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108ecc68-18ab-40e6-a004-6664e74fb746-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn\" (UID: \"108ecc68-18ab-40e6-a004-6664e74fb746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.287938 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pr9l\" (UniqueName: \"kubernetes.io/projected/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-kube-api-access-2pr9l\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns\" (UID: \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.288059 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wfs\" (UniqueName: \"kubernetes.io/projected/e475c3d4-598c-49dd-afd0-56c090b86f72-kube-api-access-w2wfs\") pod \"maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg\" (UID: \"e475c3d4-598c-49dd-afd0-56c090b86f72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.288217 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e475c3d4-598c-49dd-afd0-56c090b86f72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg\" (UID: \"e475c3d4-598c-49dd-afd0-56c090b86f72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:00.288295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.288251 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59zgn\" (UniqueName: \"kubernetes.io/projected/8b89fbf3-5d28-4308-80fc-d7d51192e711-kube-api-access-59zgn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q\" (UID: \"8b89fbf3-5d28-4308-80fc-d7d51192e711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:00.289684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.288323 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzbv\" (UniqueName: \"kubernetes.io/projected/9dab4a21-963e-4dde-9bb1-835f436ad9a9-kube-api-access-7gzbv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2\" (UID: \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:00.289684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.288394 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns\" (UID: \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:00.289684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.288424 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8w5f\" (UniqueName: \"kubernetes.io/projected/83a141e7-617e-4172-9c0c-d19df1046266-kube-api-access-b8w5f\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp\" (UID: \"83a141e7-617e-4172-9c0c-d19df1046266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:00.390204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390167 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dab4a21-963e-4dde-9bb1-835f436ad9a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2\" (UID: \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:00.390393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390217 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a141e7-617e-4172-9c0c-d19df1046266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp\" (UID: \"83a141e7-617e-4172-9c0c-d19df1046266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:00.390393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390263 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108ecc68-18ab-40e6-a004-6664e74fb746-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn\" (UID: \"108ecc68-18ab-40e6-a004-6664e74fb746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:00.390393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390297 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pr9l\" (UniqueName: \"kubernetes.io/projected/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-kube-api-access-2pr9l\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns\" (UID: \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:00.390393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390375 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e497886-fd10-4e77-8d58-8ba13a9b3c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6\" (UID: \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:00.390679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390447 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wfs\" (UniqueName: \"kubernetes.io/projected/e475c3d4-598c-49dd-afd0-56c090b86f72-kube-api-access-w2wfs\") pod \"maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg\" (UID: \"e475c3d4-598c-49dd-afd0-56c090b86f72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:00.390679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390476 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e475c3d4-598c-49dd-afd0-56c090b86f72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg\" (UID: \"e475c3d4-598c-49dd-afd0-56c090b86f72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:00.390679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390564 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpqc\" (UniqueName: \"kubernetes.io/projected/2ec07a79-6535-4a59-abef-3e1b93df3a3c-kube-api-access-jvpqc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k\" (UID: \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:00.390679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390622 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59zgn\" (UniqueName: \"kubernetes.io/projected/8b89fbf3-5d28-4308-80fc-d7d51192e711-kube-api-access-59zgn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q\" (UID: \"8b89fbf3-5d28-4308-80fc-d7d51192e711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:00.390919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390689 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzbv\" (UniqueName: \"kubernetes.io/projected/9dab4a21-963e-4dde-9bb1-835f436ad9a9-kube-api-access-7gzbv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2\" (UID: \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:00.390919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390743 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4z8\" (UniqueName: \"kubernetes.io/projected/6e497886-fd10-4e77-8d58-8ba13a9b3c89-kube-api-access-4j4z8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6\" (UID: \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:00.390919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390809 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns\" (UID: \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:00.390919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390844 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8w5f\" (UniqueName: \"kubernetes.io/projected/83a141e7-617e-4172-9c0c-d19df1046266-kube-api-access-b8w5f\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp\" (UID: \"83a141e7-617e-4172-9c0c-d19df1046266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:00.390919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390892 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w444\" (UniqueName: \"kubernetes.io/projected/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-kube-api-access-5w444\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj\" (UID: \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:00.391203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.390978 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj\" (UID: \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:00.391203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.391006 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dab4a21-963e-4dde-9bb1-835f436ad9a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2\" (UID: \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:00.391203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.391011 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108ecc68-18ab-40e6-a004-6664e74fb746-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn\" (UID: \"108ecc68-18ab-40e6-a004-6664e74fb746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:00.391203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.391050 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b89fbf3-5d28-4308-80fc-d7d51192e711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q\" (UID: \"8b89fbf3-5d28-4308-80fc-d7d51192e711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:00.391203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.391091 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a141e7-617e-4172-9c0c-d19df1046266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp\" (UID: \"83a141e7-617e-4172-9c0c-d19df1046266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:00.391401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.391354 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e475c3d4-598c-49dd-afd0-56c090b86f72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg\" (UID: \"e475c3d4-598c-49dd-afd0-56c090b86f72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:00.391439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.391392 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d879n\" (UniqueName: \"kubernetes.io/projected/108ecc68-18ab-40e6-a004-6664e74fb746-kube-api-access-d879n\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn\" (UID: \"108ecc68-18ab-40e6-a004-6664e74fb746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:00.391483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.391448 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec07a79-6535-4a59-abef-3e1b93df3a3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k\" (UID: \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:00.391537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.391492 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns\" (UID: \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:00.391882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.391860 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b89fbf3-5d28-4308-80fc-d7d51192e711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q\" (UID: \"8b89fbf3-5d28-4308-80fc-d7d51192e711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:00.414460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.414431 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59zgn\" (UniqueName: \"kubernetes.io/projected/8b89fbf3-5d28-4308-80fc-d7d51192e711-kube-api-access-59zgn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q\" (UID: \"8b89fbf3-5d28-4308-80fc-d7d51192e711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:00.417278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.417252 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pr9l\" (UniqueName: \"kubernetes.io/projected/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-kube-api-access-2pr9l\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns\" (UID: \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:00.420437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.420403 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d879n\" (UniqueName: \"kubernetes.io/projected/108ecc68-18ab-40e6-a004-6664e74fb746-kube-api-access-d879n\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn\" (UID: \"108ecc68-18ab-40e6-a004-6664e74fb746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:00.420839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.420818 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wfs\" (UniqueName: \"kubernetes.io/projected/e475c3d4-598c-49dd-afd0-56c090b86f72-kube-api-access-w2wfs\") pod \"maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg\" (UID: \"e475c3d4-598c-49dd-afd0-56c090b86f72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:00.421904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.421881 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8w5f\" (UniqueName: \"kubernetes.io/projected/83a141e7-617e-4172-9c0c-d19df1046266-kube-api-access-b8w5f\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp\" (UID: \"83a141e7-617e-4172-9c0c-d19df1046266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:00.422267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.422246 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzbv\" (UniqueName: \"kubernetes.io/projected/9dab4a21-963e-4dde-9bb1-835f436ad9a9-kube-api-access-7gzbv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2\" (UID: \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:00.481240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.481213 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:00.496034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.496006 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:00.496697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.496668 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpqc\" (UniqueName: \"kubernetes.io/projected/2ec07a79-6535-4a59-abef-3e1b93df3a3c-kube-api-access-jvpqc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k\" (UID: \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:00.496812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.496722 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4z8\" (UniqueName: \"kubernetes.io/projected/6e497886-fd10-4e77-8d58-8ba13a9b3c89-kube-api-access-4j4z8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6\" (UID: \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:00.496812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.496768 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w444\" (UniqueName: \"kubernetes.io/projected/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-kube-api-access-5w444\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj\" (UID: \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:00.496941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.496822 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj\" (UID: \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:00.496941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.496924 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec07a79-6535-4a59-abef-3e1b93df3a3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k\" (UID: \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:00.497136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.497093 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e497886-fd10-4e77-8d58-8ba13a9b3c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6\" (UID: \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:00.497785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.497760 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj\" (UID: \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:00.497896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.497862 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec07a79-6535-4a59-abef-3e1b93df3a3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k\" (UID: \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:00.497896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.497876 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e497886-fd10-4e77-8d58-8ba13a9b3c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6\" (UID: \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:00.508327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.508304 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:00.524542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.524489 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w444\" (UniqueName: \"kubernetes.io/projected/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-kube-api-access-5w444\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj\" (UID: \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:00.524809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.524740 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpqc\" (UniqueName: \"kubernetes.io/projected/2ec07a79-6535-4a59-abef-3e1b93df3a3c-kube-api-access-jvpqc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k\" (UID: \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:00.535136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.535108 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:00.536858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.536830 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4z8\" (UniqueName: \"kubernetes.io/projected/6e497886-fd10-4e77-8d58-8ba13a9b3c89-kube-api-access-4j4z8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6\" (UID: \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:00.553997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.553801 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:00.577557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.577494 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:00.595905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.594091 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:00.604278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.602924 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:00.613480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.613159 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:00.659109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.657059 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:15:00.860948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:00.856403 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg"] Oct 29 23:15:01.048401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.048368 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp"] Oct 29 23:15:01.049284 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:01.049256 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a141e7_617e_4172_9c0c_d19df1046266.slice/crio-0534b60e76a1e41b33978e7bdd78b1544cce566d9a37a16f8bad5cc7c53cd0f0 WatchSource:0}: Error finding container 0534b60e76a1e41b33978e7bdd78b1544cce566d9a37a16f8bad5cc7c53cd0f0: Status 404 returned error can't find the container with id 0534b60e76a1e41b33978e7bdd78b1544cce566d9a37a16f8bad5cc7c53cd0f0 Oct 29 23:15:01.055229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.055175 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns"] Oct 29 23:15:01.058698 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:01.058670 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1c5977_9c7c_4e88_a11c_eeed34150bbb.slice/crio-f32e5625cf0cee097aa647e5e95abb54396f0d416595f0392b88eaa9a8865d50 WatchSource:0}: Error finding container f32e5625cf0cee097aa647e5e95abb54396f0d416595f0392b88eaa9a8865d50: Status 404 returned error can't find the container with id f32e5625cf0cee097aa647e5e95abb54396f0d416595f0392b88eaa9a8865d50 Oct 29 23:15:01.123697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.123674 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2"] Oct 29 23:15:01.125275 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:01.125241 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dab4a21_963e_4dde_9bb1_835f436ad9a9.slice/crio-6c3dce155dddabce7723299965a828c7a930abd7c65b91ccd4232f14bf2a1375 WatchSource:0}: Error finding container 6c3dce155dddabce7723299965a828c7a930abd7c65b91ccd4232f14bf2a1375: Status 404 returned error can't find the container with id 6c3dce155dddabce7723299965a828c7a930abd7c65b91ccd4232f14bf2a1375 Oct 29 23:15:01.218124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.218099 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q"] Oct 29 23:15:01.242305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.242264 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn"] Oct 29 23:15:01.262778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.262756 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k"] Oct 29 23:15:01.269957 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:01.269883 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec07a79_6535_4a59_abef_3e1b93df3a3c.slice/crio-2058ecfc7b1b721a374f91b8759b1687446572fa62e5a631e3a4cf4ab33da09c WatchSource:0}: Error finding container 2058ecfc7b1b721a374f91b8759b1687446572fa62e5a631e3a4cf4ab33da09c: Status 404 returned error can't find the container with id 2058ecfc7b1b721a374f91b8759b1687446572fa62e5a631e3a4cf4ab33da09c Oct 29 23:15:01.296508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.296483 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6"] Oct 29 23:15:01.297705 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:01.297679 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e497886_fd10_4e77_8d58_8ba13a9b3c89.slice/crio-6cb674b913b7681f088b34a47d79ff7a1dc11a407dfe6d4ea2de576b7489b3b9 WatchSource:0}: Error finding container 6cb674b913b7681f088b34a47d79ff7a1dc11a407dfe6d4ea2de576b7489b3b9: Status 404 returned error can't find the container with id 6cb674b913b7681f088b34a47d79ff7a1dc11a407dfe6d4ea2de576b7489b3b9 Oct 29 23:15:01.302006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.301947 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj"] Oct 29 23:15:01.303026 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:01.302936 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16afaed_6eda_4bc6_86dc_d92e6b8c2ea3.slice/crio-e148dd59bc45c1daa5a680e7249b9a38bfa530450cde582797831ab720f6887d WatchSource:0}: Error finding container e148dd59bc45c1daa5a680e7249b9a38bfa530450cde582797831ab720f6887d: Status 404 returned error can't find the container with id e148dd59bc45c1daa5a680e7249b9a38bfa530450cde582797831ab720f6887d Oct 29 23:15:01.637931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.637892 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" event={"ID":"9dab4a21-963e-4dde-9bb1-835f436ad9a9","Type":"ContainerStarted","Data":"6c3dce155dddabce7723299965a828c7a930abd7c65b91ccd4232f14bf2a1375"} Oct 29 23:15:01.639949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.639922 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" event={"ID":"8b89fbf3-5d28-4308-80fc-d7d51192e711","Type":"ContainerStarted","Data":"5acff05c4875c7684f58d4045935e1deb065d8ec435f1ad12df6e36d8d7b938a"} Oct 29 23:15:01.642142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.642042 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" event={"ID":"108ecc68-18ab-40e6-a004-6664e74fb746","Type":"ContainerStarted","Data":"b29d6bd09cf9d96cc0927c8e2a80a67326458fd1fc1d7ed96f6a06e9a4681519"} Oct 29 23:15:01.644420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.644389 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" event={"ID":"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3","Type":"ContainerStarted","Data":"e148dd59bc45c1daa5a680e7249b9a38bfa530450cde582797831ab720f6887d"} Oct 29 23:15:01.646311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.646257 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" event={"ID":"2ec07a79-6535-4a59-abef-3e1b93df3a3c","Type":"ContainerStarted","Data":"2058ecfc7b1b721a374f91b8759b1687446572fa62e5a631e3a4cf4ab33da09c"} Oct 29 23:15:01.648163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.648137 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" event={"ID":"83a141e7-617e-4172-9c0c-d19df1046266","Type":"ContainerStarted","Data":"0534b60e76a1e41b33978e7bdd78b1544cce566d9a37a16f8bad5cc7c53cd0f0"} Oct 29 23:15:01.650069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.650049 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" event={"ID":"9c1c5977-9c7c-4e88-a11c-eeed34150bbb","Type":"ContainerStarted","Data":"b3833c634f534cab2e0f48225b484b6705ef04759f038cffc45cb8618785d1c8"} Oct 29 23:15:01.650163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.650090 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" event={"ID":"9c1c5977-9c7c-4e88-a11c-eeed34150bbb","Type":"ContainerStarted","Data":"f32e5625cf0cee097aa647e5e95abb54396f0d416595f0392b88eaa9a8865d50"} Oct 29 23:15:01.653061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.653034 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" event={"ID":"e475c3d4-598c-49dd-afd0-56c090b86f72","Type":"ContainerStarted","Data":"1bb14a786f9fa715c393d66f7de8bdde13ec25dab06d934c3ac30fb61190dfee"} Oct 29 23:15:01.653210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.653084 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" event={"ID":"e475c3d4-598c-49dd-afd0-56c090b86f72","Type":"ContainerStarted","Data":"28ebed73669313cc7db09a3cf188c1d5af1d89cd54d4a095faba0c907003f910"} Oct 29 23:15:01.656687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.656663 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" event={"ID":"6e497886-fd10-4e77-8d58-8ba13a9b3c89","Type":"ContainerStarted","Data":"6cb674b913b7681f088b34a47d79ff7a1dc11a407dfe6d4ea2de576b7489b3b9"} Oct 29 23:15:01.694130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.694067 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" podStartSLOduration=1.69404105 podStartE2EDuration="1.69404105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:01.692349475 +0000 UTC m=+1898.808269085" watchObservedRunningTime="2026-10-29 23:15:01.69404105 +0000 UTC m=+1898.809960674" Oct 29 23:15:01.738220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:01.738167 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" podStartSLOduration=1.738153065 podStartE2EDuration="1.738153065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:01.731920238 +0000 UTC m=+1898.847839825" watchObservedRunningTime="2026-10-29 23:15:01.738153065 +0000 UTC m=+1898.854072672" Oct 29 23:15:02.721546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.721513 5736 generic.go:356] "Generic (PLEG): container finished" podID="83a141e7-617e-4172-9c0c-d19df1046266" containerID="7391671a2b9c2a4d5a81bfbf5927daf4b2437fc00d7f7a1c8ec01c43934223b4" exitCode=60 Oct 29 23:15:02.722003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.721610 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" event={"ID":"83a141e7-617e-4172-9c0c-d19df1046266","Type":"ContainerDied","Data":"7391671a2b9c2a4d5a81bfbf5927daf4b2437fc00d7f7a1c8ec01c43934223b4"} Oct 29 23:15:02.722003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.721984 5736 scope.go:122] "RemoveContainer" containerID="7391671a2b9c2a4d5a81bfbf5927daf4b2437fc00d7f7a1c8ec01c43934223b4" Oct 29 23:15:02.727646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.727612 5736 generic.go:356] "Generic (PLEG): container finished" podID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerID="b3833c634f534cab2e0f48225b484b6705ef04759f038cffc45cb8618785d1c8" exitCode=60 Oct 29 23:15:02.727926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.727894 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" event={"ID":"9c1c5977-9c7c-4e88-a11c-eeed34150bbb","Type":"ContainerDied","Data":"b3833c634f534cab2e0f48225b484b6705ef04759f038cffc45cb8618785d1c8"} Oct 29 23:15:02.728255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.728237 5736 scope.go:122] "RemoveContainer" containerID="b3833c634f534cab2e0f48225b484b6705ef04759f038cffc45cb8618785d1c8" Oct 29 23:15:02.730200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.730176 5736 generic.go:356] "Generic (PLEG): container finished" podID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerID="1bb14a786f9fa715c393d66f7de8bdde13ec25dab06d934c3ac30fb61190dfee" exitCode=60 Oct 29 23:15:02.730302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.730273 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" event={"ID":"e475c3d4-598c-49dd-afd0-56c090b86f72","Type":"ContainerDied","Data":"1bb14a786f9fa715c393d66f7de8bdde13ec25dab06d934c3ac30fb61190dfee"} Oct 29 23:15:02.730690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.730557 5736 scope.go:122] "RemoveContainer" containerID="1bb14a786f9fa715c393d66f7de8bdde13ec25dab06d934c3ac30fb61190dfee" Oct 29 23:15:02.732598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.732559 5736 generic.go:356] "Generic (PLEG): container finished" podID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerID="0f7a477b8aae202acd13892f0a7bc9012550213ed05084854a247703a1d4156e" exitCode=60 Oct 29 23:15:02.732760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.732739 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" event={"ID":"6e497886-fd10-4e77-8d58-8ba13a9b3c89","Type":"ContainerDied","Data":"0f7a477b8aae202acd13892f0a7bc9012550213ed05084854a247703a1d4156e"} Oct 29 23:15:02.732935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.732917 5736 scope.go:122] "RemoveContainer" containerID="0f7a477b8aae202acd13892f0a7bc9012550213ed05084854a247703a1d4156e" Oct 29 23:15:02.734533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.734510 5736 generic.go:356] "Generic (PLEG): container finished" podID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerID="c8382f910d7c6e2a02a5282d766309fa647a5e78aa6fa45e217a6aa6b8f971d2" exitCode=60 Oct 29 23:15:02.734656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.734631 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" event={"ID":"9dab4a21-963e-4dde-9bb1-835f436ad9a9","Type":"ContainerDied","Data":"c8382f910d7c6e2a02a5282d766309fa647a5e78aa6fa45e217a6aa6b8f971d2"} Oct 29 23:15:02.734775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.734756 5736 scope.go:122] "RemoveContainer" containerID="c8382f910d7c6e2a02a5282d766309fa647a5e78aa6fa45e217a6aa6b8f971d2" Oct 29 23:15:02.736700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.736678 5736 generic.go:356] "Generic (PLEG): container finished" podID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerID="45e3a54b615e894287604a6a93403f237b7f79eb2656c6724c81790dcb0571bf" exitCode=60 Oct 29 23:15:02.736791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.736757 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" event={"ID":"8b89fbf3-5d28-4308-80fc-d7d51192e711","Type":"ContainerDied","Data":"45e3a54b615e894287604a6a93403f237b7f79eb2656c6724c81790dcb0571bf"} Oct 29 23:15:02.737032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.737015 5736 scope.go:122] "RemoveContainer" containerID="45e3a54b615e894287604a6a93403f237b7f79eb2656c6724c81790dcb0571bf" Oct 29 23:15:02.738716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.738695 5736 generic.go:356] "Generic (PLEG): container finished" podID="108ecc68-18ab-40e6-a004-6664e74fb746" containerID="e74febbda4f33ebafd45d2097c134521ab6e51e0800b06f787ace544d855a52d" exitCode=60 Oct 29 23:15:02.738802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.738774 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" event={"ID":"108ecc68-18ab-40e6-a004-6664e74fb746","Type":"ContainerDied","Data":"e74febbda4f33ebafd45d2097c134521ab6e51e0800b06f787ace544d855a52d"} Oct 29 23:15:02.738974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.738959 5736 scope.go:122] "RemoveContainer" containerID="e74febbda4f33ebafd45d2097c134521ab6e51e0800b06f787ace544d855a52d" Oct 29 23:15:02.741933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.741912 5736 generic.go:356] "Generic (PLEG): container finished" podID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerID="19bd423d038972c9acb4dd99b8492fa0ae15c4aef702b2e1bf34882e0344edd4" exitCode=60 Oct 29 23:15:02.742433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.741950 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" event={"ID":"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3","Type":"ContainerDied","Data":"19bd423d038972c9acb4dd99b8492fa0ae15c4aef702b2e1bf34882e0344edd4"} Oct 29 23:15:02.742433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.742202 5736 scope.go:122] "RemoveContainer" containerID="19bd423d038972c9acb4dd99b8492fa0ae15c4aef702b2e1bf34882e0344edd4" Oct 29 23:15:02.745246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.745224 5736 generic.go:356] "Generic (PLEG): container finished" podID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerID="162cfc7d80c227d1c8f27a2a33e6e6db0811260e45c36951397820c378e56bd2" exitCode=60 Oct 29 23:15:02.745348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.745290 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" event={"ID":"2ec07a79-6535-4a59-abef-3e1b93df3a3c","Type":"ContainerDied","Data":"162cfc7d80c227d1c8f27a2a33e6e6db0811260e45c36951397820c378e56bd2"} Oct 29 23:15:02.745559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:02.745544 5736 scope.go:122] "RemoveContainer" containerID="162cfc7d80c227d1c8f27a2a33e6e6db0811260e45c36951397820c378e56bd2" Oct 29 23:15:03.770673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.770632 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" event={"ID":"9dab4a21-963e-4dde-9bb1-835f436ad9a9","Type":"ContainerStarted","Data":"c0eb9778ba18775fae36dd18702dd6cff6247a178f34a768bf2f76fd958d8cd5"} Oct 29 23:15:03.773075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.773046 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" event={"ID":"8b89fbf3-5d28-4308-80fc-d7d51192e711","Type":"ContainerStarted","Data":"1b8d69feb0422cadec86903fab1eb210c7286e50437d7a89fa95e4dab56df806"} Oct 29 23:15:03.775174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.775151 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" event={"ID":"108ecc68-18ab-40e6-a004-6664e74fb746","Type":"ContainerStarted","Data":"d447c28d38e837683f9e8062291b159916f10afcbe499e080ff796c7df350b23"} Oct 29 23:15:03.777227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.777207 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" event={"ID":"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3","Type":"ContainerStarted","Data":"15b8b6efc10cc63594ff6dab2148ecf128bc93590a883b1f66d0df6d481a31f2"} Oct 29 23:15:03.779289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.779261 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" event={"ID":"2ec07a79-6535-4a59-abef-3e1b93df3a3c","Type":"ContainerStarted","Data":"12a510b1544fe7f87229ee2e7309cb1db0e05a7e0c47ae9dfb41b85ff35530ee"} Oct 29 23:15:03.781626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.781603 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" event={"ID":"83a141e7-617e-4172-9c0c-d19df1046266","Type":"ContainerStarted","Data":"4815dc4a39c8cb021a2eb2ededf069dbc439ef7d6a34a839d28798a4f00fc554"} Oct 29 23:15:03.783702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.783683 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" event={"ID":"9c1c5977-9c7c-4e88-a11c-eeed34150bbb","Type":"ContainerStarted","Data":"3a43086a6f760631e984441156ae8b6a5d08c19b40c7445f6663f592a2863991"} Oct 29 23:15:03.785618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.785566 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" event={"ID":"e475c3d4-598c-49dd-afd0-56c090b86f72","Type":"ContainerStarted","Data":"07b30aa614d42ec4cfa100a63356f724b08a17226a3391f74ff5018ecacbc0ad"} Oct 29 23:15:03.787297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.787278 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" event={"ID":"6e497886-fd10-4e77-8d58-8ba13a9b3c89","Type":"ContainerStarted","Data":"f99abe781d43eedcc77069e4beaa3f0cd1b7889e836dd7b29998d660be406d86"} Oct 29 23:15:03.812955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.812874 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" podStartSLOduration=3.812861783 podStartE2EDuration="3.812861783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.810023124 +0000 UTC m=+1900.925942731" watchObservedRunningTime="2026-10-29 23:15:03.812861783 +0000 UTC m=+1900.928781389" Oct 29 23:15:03.889871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.889819 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" podStartSLOduration=3.8897935070000003 podStartE2EDuration="3.889793507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.888547235 +0000 UTC m=+1901.004466848" watchObservedRunningTime="2026-10-29 23:15:03.889793507 +0000 UTC m=+1901.005713114" Oct 29 23:15:03.979022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:03.978969 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" podStartSLOduration=3.978952139 podStartE2EDuration="3.978952139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.973080647 +0000 UTC m=+1901.089000355" watchObservedRunningTime="2026-10-29 23:15:03.978952139 +0000 UTC m=+1901.094871745" Oct 29 23:15:04.015312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.015251 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" podStartSLOduration=4.015235505 podStartE2EDuration="4.015235505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.012152889 +0000 UTC m=+1901.128072495" watchObservedRunningTime="2026-10-29 23:15:04.015235505 +0000 UTC m=+1901.131155110" Oct 29 23:15:04.055764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.055711 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" podStartSLOduration=4.055696472 podStartE2EDuration="4.055696472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.051087378 +0000 UTC m=+1901.167006984" watchObservedRunningTime="2026-10-29 23:15:04.055696472 +0000 UTC m=+1901.171616075" Oct 29 23:15:04.093294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.093237 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" podStartSLOduration=4.093219068 podStartE2EDuration="4.093219068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.092364007 +0000 UTC m=+1901.208283613" watchObservedRunningTime="2026-10-29 23:15:04.093219068 +0000 UTC m=+1901.209138689" Oct 29 23:15:04.138219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.138005 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" podStartSLOduration=4.137990057 podStartE2EDuration="4.137990057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.133062728 +0000 UTC m=+1901.248982333" watchObservedRunningTime="2026-10-29 23:15:04.137990057 +0000 UTC m=+1901.253909722" Oct 29 23:15:04.800575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.800543 5736 generic.go:356] "Generic (PLEG): container finished" podID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerID="1b8d69feb0422cadec86903fab1eb210c7286e50437d7a89fa95e4dab56df806" exitCode=60 Oct 29 23:15:04.801084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.800626 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" event={"ID":"8b89fbf3-5d28-4308-80fc-d7d51192e711","Type":"ContainerDied","Data":"1b8d69feb0422cadec86903fab1eb210c7286e50437d7a89fa95e4dab56df806"} Oct 29 23:15:04.801084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.800690 5736 scope.go:122] "RemoveContainer" containerID="45e3a54b615e894287604a6a93403f237b7f79eb2656c6724c81790dcb0571bf" Oct 29 23:15:04.801084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.801070 5736 scope.go:122] "RemoveContainer" containerID="1b8d69feb0422cadec86903fab1eb210c7286e50437d7a89fa95e4dab56df806" Oct 29 23:15:04.801507 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:04.801482 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q_odh-ai-gateway-infra(8b89fbf3-5d28-4308-80fc-d7d51192e711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" Oct 29 23:15:04.802980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.802959 5736 generic.go:356] "Generic (PLEG): container finished" podID="108ecc68-18ab-40e6-a004-6664e74fb746" containerID="d447c28d38e837683f9e8062291b159916f10afcbe499e080ff796c7df350b23" exitCode=60 Oct 29 23:15:04.803113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.803035 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" event={"ID":"108ecc68-18ab-40e6-a004-6664e74fb746","Type":"ContainerDied","Data":"d447c28d38e837683f9e8062291b159916f10afcbe499e080ff796c7df350b23"} Oct 29 23:15:04.803312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.803296 5736 scope.go:122] "RemoveContainer" containerID="d447c28d38e837683f9e8062291b159916f10afcbe499e080ff796c7df350b23" Oct 29 23:15:04.803693 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:04.803674 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn_odh-ai-gateway-infra(108ecc68-18ab-40e6-a004-6664e74fb746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" Oct 29 23:15:04.805160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.805136 5736 generic.go:356] "Generic (PLEG): container finished" podID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerID="15b8b6efc10cc63594ff6dab2148ecf128bc93590a883b1f66d0df6d481a31f2" exitCode=60 Oct 29 23:15:04.805246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.805207 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" event={"ID":"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3","Type":"ContainerDied","Data":"15b8b6efc10cc63594ff6dab2148ecf128bc93590a883b1f66d0df6d481a31f2"} Oct 29 23:15:04.805484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.805463 5736 scope.go:122] "RemoveContainer" containerID="15b8b6efc10cc63594ff6dab2148ecf128bc93590a883b1f66d0df6d481a31f2" Oct 29 23:15:04.805834 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:04.805813 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj_odh-ai-gateway-infra(b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" Oct 29 23:15:04.807313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.807289 5736 generic.go:356] "Generic (PLEG): container finished" podID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerID="12a510b1544fe7f87229ee2e7309cb1db0e05a7e0c47ae9dfb41b85ff35530ee" exitCode=60 Oct 29 23:15:04.807425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.807408 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" event={"ID":"2ec07a79-6535-4a59-abef-3e1b93df3a3c","Type":"ContainerDied","Data":"12a510b1544fe7f87229ee2e7309cb1db0e05a7e0c47ae9dfb41b85ff35530ee"} Oct 29 23:15:04.807608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.807574 5736 scope.go:122] "RemoveContainer" containerID="12a510b1544fe7f87229ee2e7309cb1db0e05a7e0c47ae9dfb41b85ff35530ee" Oct 29 23:15:04.807950 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:04.807922 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k_odh-ai-gateway-infra(2ec07a79-6535-4a59-abef-3e1b93df3a3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" Oct 29 23:15:04.809323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.809304 5736 generic.go:356] "Generic (PLEG): container finished" podID="83a141e7-617e-4172-9c0c-d19df1046266" containerID="4815dc4a39c8cb021a2eb2ededf069dbc439ef7d6a34a839d28798a4f00fc554" exitCode=60 Oct 29 23:15:04.809550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.809522 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" event={"ID":"83a141e7-617e-4172-9c0c-d19df1046266","Type":"ContainerDied","Data":"4815dc4a39c8cb021a2eb2ededf069dbc439ef7d6a34a839d28798a4f00fc554"} Oct 29 23:15:04.811212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.811182 5736 generic.go:356] "Generic (PLEG): container finished" podID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerID="3a43086a6f760631e984441156ae8b6a5d08c19b40c7445f6663f592a2863991" exitCode=60 Oct 29 23:15:04.811288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.811273 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" event={"ID":"9c1c5977-9c7c-4e88-a11c-eeed34150bbb","Type":"ContainerDied","Data":"3a43086a6f760631e984441156ae8b6a5d08c19b40c7445f6663f592a2863991"} Oct 29 23:15:04.811720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.811703 5736 scope.go:122] "RemoveContainer" containerID="4815dc4a39c8cb021a2eb2ededf069dbc439ef7d6a34a839d28798a4f00fc554" Oct 29 23:15:04.812017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.812000 5736 scope.go:122] "RemoveContainer" containerID="3a43086a6f760631e984441156ae8b6a5d08c19b40c7445f6663f592a2863991" Oct 29 23:15:04.812017 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:04.812006 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp_odh-ai-gateway-infra(83a141e7-617e-4172-9c0c-d19df1046266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" podUID="83a141e7-617e-4172-9c0c-d19df1046266" Oct 29 23:15:04.812284 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:04.812260 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns_odh-ai-gateway-infra(9c1c5977-9c7c-4e88-a11c-eeed34150bbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" Oct 29 23:15:04.813326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.813304 5736 generic.go:356] "Generic (PLEG): container finished" podID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerID="07b30aa614d42ec4cfa100a63356f724b08a17226a3391f74ff5018ecacbc0ad" exitCode=60 Oct 29 23:15:04.813403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.813389 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" event={"ID":"e475c3d4-598c-49dd-afd0-56c090b86f72","Type":"ContainerDied","Data":"07b30aa614d42ec4cfa100a63356f724b08a17226a3391f74ff5018ecacbc0ad"} Oct 29 23:15:04.813744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.813726 5736 scope.go:122] "RemoveContainer" containerID="07b30aa614d42ec4cfa100a63356f724b08a17226a3391f74ff5018ecacbc0ad" Oct 29 23:15:04.814042 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:04.814020 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg_odh-ai-gateway-infra(e475c3d4-598c-49dd-afd0-56c090b86f72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" Oct 29 23:15:04.815587 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.815563 5736 generic.go:356] "Generic (PLEG): container finished" podID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerID="f99abe781d43eedcc77069e4beaa3f0cd1b7889e836dd7b29998d660be406d86" exitCode=60 Oct 29 23:15:04.815695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.815637 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" event={"ID":"6e497886-fd10-4e77-8d58-8ba13a9b3c89","Type":"ContainerDied","Data":"f99abe781d43eedcc77069e4beaa3f0cd1b7889e836dd7b29998d660be406d86"} Oct 29 23:15:04.815911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.815896 5736 scope.go:122] "RemoveContainer" containerID="f99abe781d43eedcc77069e4beaa3f0cd1b7889e836dd7b29998d660be406d86" Oct 29 23:15:04.816215 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:04.816191 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6_odh-ai-gateway-infra(6e497886-fd10-4e77-8d58-8ba13a9b3c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" Oct 29 23:15:04.816919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.816901 5736 scope.go:122] "RemoveContainer" containerID="e74febbda4f33ebafd45d2097c134521ab6e51e0800b06f787ace544d855a52d" Oct 29 23:15:04.817877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.817754 5736 generic.go:356] "Generic (PLEG): container finished" podID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerID="c0eb9778ba18775fae36dd18702dd6cff6247a178f34a768bf2f76fd958d8cd5" exitCode=60 Oct 29 23:15:04.817877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.817803 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" event={"ID":"9dab4a21-963e-4dde-9bb1-835f436ad9a9","Type":"ContainerDied","Data":"c0eb9778ba18775fae36dd18702dd6cff6247a178f34a768bf2f76fd958d8cd5"} Oct 29 23:15:04.818034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.818020 5736 scope.go:122] "RemoveContainer" containerID="c0eb9778ba18775fae36dd18702dd6cff6247a178f34a768bf2f76fd958d8cd5" Oct 29 23:15:04.818257 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:04.818240 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2_odh-ai-gateway-infra(9dab4a21-963e-4dde-9bb1-835f436ad9a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" Oct 29 23:15:04.831345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.831325 5736 scope.go:122] "RemoveContainer" containerID="19bd423d038972c9acb4dd99b8492fa0ae15c4aef702b2e1bf34882e0344edd4" Oct 29 23:15:04.845840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.845821 5736 scope.go:122] "RemoveContainer" containerID="162cfc7d80c227d1c8f27a2a33e6e6db0811260e45c36951397820c378e56bd2" Oct 29 23:15:04.859957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.859938 5736 scope.go:122] "RemoveContainer" containerID="7391671a2b9c2a4d5a81bfbf5927daf4b2437fc00d7f7a1c8ec01c43934223b4" Oct 29 23:15:04.874265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.874237 5736 scope.go:122] "RemoveContainer" containerID="b3833c634f534cab2e0f48225b484b6705ef04759f038cffc45cb8618785d1c8" Oct 29 23:15:04.888778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.888760 5736 scope.go:122] "RemoveContainer" containerID="1bb14a786f9fa715c393d66f7de8bdde13ec25dab06d934c3ac30fb61190dfee" Oct 29 23:15:04.902758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.902729 5736 scope.go:122] "RemoveContainer" containerID="0f7a477b8aae202acd13892f0a7bc9012550213ed05084854a247703a1d4156e" Oct 29 23:15:04.920306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:04.920283 5736 scope.go:122] "RemoveContainer" containerID="c8382f910d7c6e2a02a5282d766309fa647a5e78aa6fa45e217a6aa6b8f971d2" Oct 29 23:15:05.828821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:05.828790 5736 scope.go:122] "RemoveContainer" containerID="3a43086a6f760631e984441156ae8b6a5d08c19b40c7445f6663f592a2863991" Oct 29 23:15:05.829314 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:05.829213 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns_odh-ai-gateway-infra(9c1c5977-9c7c-4e88-a11c-eeed34150bbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" Oct 29 23:15:05.830631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:05.830607 5736 scope.go:122] "RemoveContainer" containerID="07b30aa614d42ec4cfa100a63356f724b08a17226a3391f74ff5018ecacbc0ad" Oct 29 23:15:05.830941 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:05.830917 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg_odh-ai-gateway-infra(e475c3d4-598c-49dd-afd0-56c090b86f72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" Oct 29 23:15:05.832223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:05.832197 5736 scope.go:122] "RemoveContainer" containerID="f99abe781d43eedcc77069e4beaa3f0cd1b7889e836dd7b29998d660be406d86" Oct 29 23:15:05.832498 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:05.832475 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6_odh-ai-gateway-infra(6e497886-fd10-4e77-8d58-8ba13a9b3c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" Oct 29 23:15:05.833783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:05.833763 5736 scope.go:122] "RemoveContainer" containerID="c0eb9778ba18775fae36dd18702dd6cff6247a178f34a768bf2f76fd958d8cd5" Oct 29 23:15:05.834048 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:05.834025 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2_odh-ai-gateway-infra(9dab4a21-963e-4dde-9bb1-835f436ad9a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" Oct 29 23:15:05.835375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:05.835358 5736 scope.go:122] "RemoveContainer" containerID="1b8d69feb0422cadec86903fab1eb210c7286e50437d7a89fa95e4dab56df806" Oct 29 23:15:05.835676 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:05.835653 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q_odh-ai-gateway-infra(8b89fbf3-5d28-4308-80fc-d7d51192e711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" Oct 29 23:15:05.837092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:05.837074 5736 scope.go:122] "RemoveContainer" containerID="d447c28d38e837683f9e8062291b159916f10afcbe499e080ff796c7df350b23" Oct 29 23:15:05.837351 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:05.837331 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn_odh-ai-gateway-infra(108ecc68-18ab-40e6-a004-6664e74fb746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" Oct 29 23:15:05.838857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:05.838839 5736 scope.go:122] "RemoveContainer" containerID="15b8b6efc10cc63594ff6dab2148ecf128bc93590a883b1f66d0df6d481a31f2" Oct 29 23:15:05.839102 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:05.839082 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj_odh-ai-gateway-infra(b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" Oct 29 23:15:05.840619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:05.840601 5736 scope.go:122] "RemoveContainer" containerID="12a510b1544fe7f87229ee2e7309cb1db0e05a7e0c47ae9dfb41b85ff35530ee" Oct 29 23:15:05.840852 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:05.840828 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k_odh-ai-gateway-infra(2ec07a79-6535-4a59-abef-3e1b93df3a3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" Oct 29 23:15:05.842368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:05.842351 5736 scope.go:122] "RemoveContainer" containerID="4815dc4a39c8cb021a2eb2ededf069dbc439ef7d6a34a839d28798a4f00fc554" Oct 29 23:15:05.842564 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:05.842548 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp_odh-ai-gateway-infra(83a141e7-617e-4172-9c0c-d19df1046266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" podUID="83a141e7-617e-4172-9c0c-d19df1046266" Oct 29 23:15:14.996454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:14.996411 5736 scope.go:122] "RemoveContainer" containerID="12a510b1544fe7f87229ee2e7309cb1db0e05a7e0c47ae9dfb41b85ff35530ee" Oct 29 23:15:14.996985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:14.996844 5736 scope.go:122] "RemoveContainer" containerID="1b8d69feb0422cadec86903fab1eb210c7286e50437d7a89fa95e4dab56df806" Oct 29 23:15:15.997245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:15.997201 5736 scope.go:122] "RemoveContainer" containerID="3a43086a6f760631e984441156ae8b6a5d08c19b40c7445f6663f592a2863991" Oct 29 23:15:15.999538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:15.999511 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" event={"ID":"8b89fbf3-5d28-4308-80fc-d7d51192e711","Type":"ContainerStarted","Data":"f8742d6a60d038edd825ec3744ba5e135e4f6a11254f219c7d0d86f9a7a7cc77"} Oct 29 23:15:16.001723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:16.001698 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" event={"ID":"2ec07a79-6535-4a59-abef-3e1b93df3a3c","Type":"ContainerStarted","Data":"e047586e2f0e780ce37d04365591e1c2ebbf1f22ec502824d1296bfd53fb4859"} Oct 29 23:15:17.024600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.023766 5736 scope.go:122] "RemoveContainer" containerID="f99abe781d43eedcc77069e4beaa3f0cd1b7889e836dd7b29998d660be406d86" Oct 29 23:15:17.047820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.046135 5736 scope.go:122] "RemoveContainer" containerID="07b30aa614d42ec4cfa100a63356f724b08a17226a3391f74ff5018ecacbc0ad" Oct 29 23:15:17.203106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.203072 5736 generic.go:356] "Generic (PLEG): container finished" podID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerID="f8742d6a60d038edd825ec3744ba5e135e4f6a11254f219c7d0d86f9a7a7cc77" exitCode=60 Oct 29 23:15:17.203222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.203177 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" event={"ID":"8b89fbf3-5d28-4308-80fc-d7d51192e711","Type":"ContainerDied","Data":"f8742d6a60d038edd825ec3744ba5e135e4f6a11254f219c7d0d86f9a7a7cc77"} Oct 29 23:15:17.203284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.203233 5736 scope.go:122] "RemoveContainer" containerID="1b8d69feb0422cadec86903fab1eb210c7286e50437d7a89fa95e4dab56df806" Oct 29 23:15:17.203804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.203782 5736 scope.go:122] "RemoveContainer" containerID="f8742d6a60d038edd825ec3744ba5e135e4f6a11254f219c7d0d86f9a7a7cc77" Oct 29 23:15:17.204255 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:17.204223 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q_odh-ai-gateway-infra(8b89fbf3-5d28-4308-80fc-d7d51192e711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" Oct 29 23:15:17.213280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.213211 5736 generic.go:356] "Generic (PLEG): container finished" podID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerID="e047586e2f0e780ce37d04365591e1c2ebbf1f22ec502824d1296bfd53fb4859" exitCode=60 Oct 29 23:15:17.213770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.213717 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" event={"ID":"2ec07a79-6535-4a59-abef-3e1b93df3a3c","Type":"ContainerDied","Data":"e047586e2f0e780ce37d04365591e1c2ebbf1f22ec502824d1296bfd53fb4859"} Oct 29 23:15:17.214553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.214477 5736 scope.go:122] "RemoveContainer" containerID="e047586e2f0e780ce37d04365591e1c2ebbf1f22ec502824d1296bfd53fb4859" Oct 29 23:15:17.215310 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:17.215240 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k_odh-ai-gateway-infra(2ec07a79-6535-4a59-abef-3e1b93df3a3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" Oct 29 23:15:17.215758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.215656 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q"] Oct 29 23:15:17.220603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.219836 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" event={"ID":"9c1c5977-9c7c-4e88-a11c-eeed34150bbb","Type":"ContainerStarted","Data":"6060f2353ae7b439b548d5a65a07d609803f7dbc9113fa00425033e627f9b00e"} Oct 29 23:15:17.291155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.291118 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k"] Oct 29 23:15:17.329310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.329287 5736 scope.go:122] "RemoveContainer" containerID="12a510b1544fe7f87229ee2e7309cb1db0e05a7e0c47ae9dfb41b85ff35530ee" Oct 29 23:15:17.996656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.996615 5736 scope.go:122] "RemoveContainer" containerID="c0eb9778ba18775fae36dd18702dd6cff6247a178f34a768bf2f76fd958d8cd5" Oct 29 23:15:17.997031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.996991 5736 scope.go:122] "RemoveContainer" containerID="4815dc4a39c8cb021a2eb2ededf069dbc439ef7d6a34a839d28798a4f00fc554" Oct 29 23:15:17.997287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.997258 5736 scope.go:122] "RemoveContainer" containerID="d447c28d38e837683f9e8062291b159916f10afcbe499e080ff796c7df350b23" Oct 29 23:15:17.997622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:17.997564 5736 scope.go:122] "RemoveContainer" containerID="15b8b6efc10cc63594ff6dab2148ecf128bc93590a883b1f66d0df6d481a31f2" Oct 29 23:15:18.266464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.266429 5736 generic.go:356] "Generic (PLEG): container finished" podID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerID="6060f2353ae7b439b548d5a65a07d609803f7dbc9113fa00425033e627f9b00e" exitCode=60 Oct 29 23:15:18.266938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.266570 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" event={"ID":"9c1c5977-9c7c-4e88-a11c-eeed34150bbb","Type":"ContainerDied","Data":"6060f2353ae7b439b548d5a65a07d609803f7dbc9113fa00425033e627f9b00e"} Oct 29 23:15:18.266938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.266647 5736 scope.go:122] "RemoveContainer" containerID="3a43086a6f760631e984441156ae8b6a5d08c19b40c7445f6663f592a2863991" Oct 29 23:15:18.267402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.267215 5736 scope.go:122] "RemoveContainer" containerID="6060f2353ae7b439b548d5a65a07d609803f7dbc9113fa00425033e627f9b00e" Oct 29 23:15:18.268866 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:18.267832 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns_odh-ai-gateway-infra(9c1c5977-9c7c-4e88-a11c-eeed34150bbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" Oct 29 23:15:18.273192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.273164 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" event={"ID":"e475c3d4-598c-49dd-afd0-56c090b86f72","Type":"ContainerStarted","Data":"c1114aa163e8dffabdcc68d26426c6a1fedba2ddbcd2c927a4b8c893c1db4bb3"} Oct 29 23:15:18.278543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.278505 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" event={"ID":"6e497886-fd10-4e77-8d58-8ba13a9b3c89","Type":"ContainerStarted","Data":"62ccd5db1710cb72644c1f495361805992353494f88e39c9a4019b65dac0a63b"} Oct 29 23:15:18.320253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.320220 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns"] Oct 29 23:15:18.686683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.686653 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:18.691530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.691507 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:18.757241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.755562 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b89fbf3-5d28-4308-80fc-d7d51192e711-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b89fbf3-5d28-4308-80fc-d7d51192e711-maas-api-service-ca\") pod \"8b89fbf3-5d28-4308-80fc-d7d51192e711\" (UID: \"8b89fbf3-5d28-4308-80fc-d7d51192e711\") " Oct 29 23:15:18.757241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.755640 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec07a79-6535-4a59-abef-3e1b93df3a3c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec07a79-6535-4a59-abef-3e1b93df3a3c-maas-api-service-ca\") pod \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\" (UID: \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\") " Oct 29 23:15:18.757241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.755736 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec07a79-6535-4a59-abef-3e1b93df3a3c-kube-api-access-jvpqc\" (UniqueName: \"kubernetes.io/projected/2ec07a79-6535-4a59-abef-3e1b93df3a3c-kube-api-access-jvpqc\") pod \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\" (UID: \"2ec07a79-6535-4a59-abef-3e1b93df3a3c\") " Oct 29 23:15:18.757241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.755862 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b89fbf3-5d28-4308-80fc-d7d51192e711-kube-api-access-59zgn\" (UniqueName: \"kubernetes.io/projected/8b89fbf3-5d28-4308-80fc-d7d51192e711-kube-api-access-59zgn\") pod \"8b89fbf3-5d28-4308-80fc-d7d51192e711\" (UID: \"8b89fbf3-5d28-4308-80fc-d7d51192e711\") " Oct 29 23:15:18.758650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.758563 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec07a79-6535-4a59-abef-3e1b93df3a3c-maas-api-service-ca" pod "2ec07a79-6535-4a59-abef-3e1b93df3a3c" (UID: "2ec07a79-6535-4a59-abef-3e1b93df3a3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:18.758958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.758937 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b89fbf3-5d28-4308-80fc-d7d51192e711-maas-api-service-ca" pod "8b89fbf3-5d28-4308-80fc-d7d51192e711" (UID: "8b89fbf3-5d28-4308-80fc-d7d51192e711"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:18.761378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.761350 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec07a79-6535-4a59-abef-3e1b93df3a3c-kube-api-access-jvpqc" pod "2ec07a79-6535-4a59-abef-3e1b93df3a3c" (UID: "2ec07a79-6535-4a59-abef-3e1b93df3a3c"). InnerVolumeSpecName "kube-api-access-jvpqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:18.786655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.786604 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b89fbf3-5d28-4308-80fc-d7d51192e711-kube-api-access-59zgn" pod "8b89fbf3-5d28-4308-80fc-d7d51192e711" (UID: "8b89fbf3-5d28-4308-80fc-d7d51192e711"). InnerVolumeSpecName "kube-api-access-59zgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:18.858821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.858766 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpqc\" (UniqueName: \"kubernetes.io/projected/2ec07a79-6535-4a59-abef-3e1b93df3a3c-kube-api-access-jvpqc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:18.858982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.858816 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59zgn\" (UniqueName: \"kubernetes.io/projected/8b89fbf3-5d28-4308-80fc-d7d51192e711-kube-api-access-59zgn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:18.858982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.858848 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b89fbf3-5d28-4308-80fc-d7d51192e711-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:18.858982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.858865 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ec07a79-6535-4a59-abef-3e1b93df3a3c-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:18.936043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.935972 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4"] Oct 29 23:15:18.936801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936781 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerName="cleanup" Oct 29 23:15:18.936856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936804 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerName="cleanup" Oct 29 23:15:18.936856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936818 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerName="cleanup" Oct 29 23:15:18.936856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936824 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerName="cleanup" Oct 29 23:15:18.936970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936863 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerName="cleanup" Oct 29 23:15:18.936970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936870 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerName="cleanup" Oct 29 23:15:18.936970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936891 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerName="cleanup" Oct 29 23:15:18.936970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936898 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerName="cleanup" Oct 29 23:15:18.936970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936914 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerName="cleanup" Oct 29 23:15:18.936970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.936920 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerName="cleanup" Oct 29 23:15:18.937188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.937103 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerName="cleanup" Oct 29 23:15:18.937188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.937115 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerName="cleanup" Oct 29 23:15:18.937188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.937138 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerName="cleanup" Oct 29 23:15:18.937188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.937148 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerName="cleanup" Oct 29 23:15:18.941427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.941394 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:18.970190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.963317 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c47a72\"/\"explicit-model-c47a72-kserve-self-signed-certs\"" Oct 29 23:15:18.970190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.967723 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c47a72\"/\"kube-root-ca.crt\"" Oct 29 23:15:18.970190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.967805 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c47a72\"/\"default-dockercfg-jr6kn\"" Oct 29 23:15:18.970190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.967914 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c47a72\"/\"openshift-service-ca.crt\"" Oct 29 23:15:18.970658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:18.970632 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4"] Oct 29 23:15:19.062128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.062082 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tls-certs\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.062327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.062294 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-model-cache\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.062417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.062348 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tmp-dir\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.062482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.062414 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-home\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.062547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.062501 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82vr\" (UniqueName: \"kubernetes.io/projected/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-kube-api-access-g82vr\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.062847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.062822 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-dshm\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.167687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.167125 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tls-certs\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.169313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.169269 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-model-cache\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.169492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.169329 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tmp-dir\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.169492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.169394 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-home\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.169492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.169483 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82vr\" (UniqueName: \"kubernetes.io/projected/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-kube-api-access-g82vr\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.180835 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.174228 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-model-cache\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.202646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.197148 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-dshm\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.202646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.198039 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tmp-dir\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.202646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.198784 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-home\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.209629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.203070 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-dshm\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.220614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.220513 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tls-certs\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.231615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.231538 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82vr\" (UniqueName: \"kubernetes.io/projected/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-kube-api-access-g82vr\") pod \"explicit-model-c47a72-kserve-5cff444c55-ck4w4\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.275605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.273015 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:19.384966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.384922 5736 generic.go:356] "Generic (PLEG): container finished" podID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerID="df14cba9107b57a871c00f0cf8c7198a5cd6e95eaaf0b6d10b37713e941ae757" exitCode=60 Oct 29 23:15:19.385178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.385151 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" event={"ID":"9dab4a21-963e-4dde-9bb1-835f436ad9a9","Type":"ContainerDied","Data":"df14cba9107b57a871c00f0cf8c7198a5cd6e95eaaf0b6d10b37713e941ae757"} Oct 29 23:15:19.385267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.385221 5736 scope.go:122] "RemoveContainer" containerID="c0eb9778ba18775fae36dd18702dd6cff6247a178f34a768bf2f76fd958d8cd5" Oct 29 23:15:19.386340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.386318 5736 scope.go:122] "RemoveContainer" containerID="df14cba9107b57a871c00f0cf8c7198a5cd6e95eaaf0b6d10b37713e941ae757" Oct 29 23:15:19.387446 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:19.387405 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2_odh-ai-gateway-infra(9dab4a21-963e-4dde-9bb1-835f436ad9a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" Oct 29 23:15:19.388358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.388327 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" event={"ID":"8b89fbf3-5d28-4308-80fc-d7d51192e711","Type":"ContainerDied","Data":"5acff05c4875c7684f58d4045935e1deb065d8ec435f1ad12df6e36d8d7b938a"} Oct 29 23:15:19.397641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.393865 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q" Oct 29 23:15:19.405615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.405077 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" event={"ID":"108ecc68-18ab-40e6-a004-6664e74fb746","Type":"ContainerStarted","Data":"1e600d21f4a1f22c1d753a2045d1c5f65490e0e285e7e81bf8f25d336f6cde9d"} Oct 29 23:15:19.405763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.405620 5736 scope.go:122] "RemoveContainer" containerID="1e600d21f4a1f22c1d753a2045d1c5f65490e0e285e7e81bf8f25d336f6cde9d" Oct 29 23:15:19.406119 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:19.406047 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn_odh-ai-gateway-infra(108ecc68-18ab-40e6-a004-6664e74fb746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" Oct 29 23:15:19.420144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.409900 5736 generic.go:356] "Generic (PLEG): container finished" podID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerID="d78669ddbba7ced3b4a43d5348ee0bb717471b69d67b8c6b05feb3caf8cbbdcc" exitCode=60 Oct 29 23:15:19.420144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.409981 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" event={"ID":"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3","Type":"ContainerDied","Data":"d78669ddbba7ced3b4a43d5348ee0bb717471b69d67b8c6b05feb3caf8cbbdcc"} Oct 29 23:15:19.420144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.410367 5736 scope.go:122] "RemoveContainer" containerID="d78669ddbba7ced3b4a43d5348ee0bb717471b69d67b8c6b05feb3caf8cbbdcc" Oct 29 23:15:19.420144 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:19.410679 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj_odh-ai-gateway-infra(b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" Oct 29 23:15:19.420144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.414204 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" event={"ID":"2ec07a79-6535-4a59-abef-3e1b93df3a3c","Type":"ContainerDied","Data":"2058ecfc7b1b721a374f91b8759b1687446572fa62e5a631e3a4cf4ab33da09c"} Oct 29 23:15:19.420144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.414300 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k" Oct 29 23:15:19.427044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.423143 5736 generic.go:356] "Generic (PLEG): container finished" podID="83a141e7-617e-4172-9c0c-d19df1046266" containerID="f33f64793fb8eaedc08e2787e7f0b34a4ab595ac483b1f97cf11fd218975c5fc" exitCode=60 Oct 29 23:15:19.427044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.423274 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" event={"ID":"83a141e7-617e-4172-9c0c-d19df1046266","Type":"ContainerDied","Data":"f33f64793fb8eaedc08e2787e7f0b34a4ab595ac483b1f97cf11fd218975c5fc"} Oct 29 23:15:19.427044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.423710 5736 scope.go:122] "RemoveContainer" containerID="f33f64793fb8eaedc08e2787e7f0b34a4ab595ac483b1f97cf11fd218975c5fc" Oct 29 23:15:19.427044 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:19.424017 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp_odh-ai-gateway-infra(83a141e7-617e-4172-9c0c-d19df1046266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" podUID="83a141e7-617e-4172-9c0c-d19df1046266" Oct 29 23:15:19.427044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.424082 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg"] Oct 29 23:15:19.436081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.436052 5736 generic.go:356] "Generic (PLEG): container finished" podID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerID="c1114aa163e8dffabdcc68d26426c6a1fedba2ddbcd2c927a4b8c893c1db4bb3" exitCode=60 Oct 29 23:15:19.436202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.436147 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" event={"ID":"e475c3d4-598c-49dd-afd0-56c090b86f72","Type":"ContainerDied","Data":"c1114aa163e8dffabdcc68d26426c6a1fedba2ddbcd2c927a4b8c893c1db4bb3"} Oct 29 23:15:19.464243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.461452 5736 scope.go:122] "RemoveContainer" containerID="f8742d6a60d038edd825ec3744ba5e135e4f6a11254f219c7d0d86f9a7a7cc77" Oct 29 23:15:19.464868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.464726 5736 generic.go:356] "Generic (PLEG): container finished" podID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerID="62ccd5db1710cb72644c1f495361805992353494f88e39c9a4019b65dac0a63b" exitCode=60 Oct 29 23:15:19.464868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.464789 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" event={"ID":"6e497886-fd10-4e77-8d58-8ba13a9b3c89","Type":"ContainerDied","Data":"62ccd5db1710cb72644c1f495361805992353494f88e39c9a4019b65dac0a63b"} Oct 29 23:15:19.466897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.465814 5736 scope.go:122] "RemoveContainer" containerID="62ccd5db1710cb72644c1f495361805992353494f88e39c9a4019b65dac0a63b" Oct 29 23:15:19.466897 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:19.466263 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6_odh-ai-gateway-infra(6e497886-fd10-4e77-8d58-8ba13a9b3c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" Oct 29 23:15:19.472407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.472218 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6"] Oct 29 23:15:19.524015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.523982 5736 scope.go:122] "RemoveContainer" containerID="15b8b6efc10cc63594ff6dab2148ecf128bc93590a883b1f66d0df6d481a31f2" Oct 29 23:15:19.572914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.572823 5736 scope.go:122] "RemoveContainer" containerID="e047586e2f0e780ce37d04365591e1c2ebbf1f22ec502824d1296bfd53fb4859" Oct 29 23:15:19.590339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.590235 5736 scope.go:122] "RemoveContainer" containerID="4815dc4a39c8cb021a2eb2ededf069dbc439ef7d6a34a839d28798a4f00fc554" Oct 29 23:15:19.614101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.614078 5736 scope.go:122] "RemoveContainer" containerID="07b30aa614d42ec4cfa100a63356f724b08a17226a3391f74ff5018ecacbc0ad" Oct 29 23:15:19.640096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.640063 5736 scope.go:122] "RemoveContainer" containerID="f99abe781d43eedcc77069e4beaa3f0cd1b7889e836dd7b29998d660be406d86" Oct 29 23:15:19.699081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.699046 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q"] Oct 29 23:15:19.711215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.711179 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888595-9mc2q"] Oct 29 23:15:19.808363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.808336 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:19.810898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.810870 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:19.836939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.836913 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4"] Oct 29 23:15:19.837518 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:19.837396 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5508227_0dc5_4a0d_819f_fb2a6f8a2ec3.slice/crio-69132e3f676fec56e23f5d47452b67c140c67c888d752f0dfc75f28502e75c76 WatchSource:0}: Error finding container 69132e3f676fec56e23f5d47452b67c140c67c888d752f0dfc75f28502e75c76: Status 404 returned error can't find the container with id 69132e3f676fec56e23f5d47452b67c140c67c888d752f0dfc75f28502e75c76 Oct 29 23:15:19.848216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.848189 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k"] Oct 29 23:15:19.866693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.866650 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888595-sh99k"] Oct 29 23:15:19.917138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.917103 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e475c3d4-598c-49dd-afd0-56c090b86f72-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e475c3d4-598c-49dd-afd0-56c090b86f72-maas-api-service-ca\") pod \"e475c3d4-598c-49dd-afd0-56c090b86f72\" (UID: \"e475c3d4-598c-49dd-afd0-56c090b86f72\") " Oct 29 23:15:19.917566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.917542 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-maas-api-service-ca\") pod \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\" (UID: \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\") " Oct 29 23:15:19.917714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.917655 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-kube-api-access-2pr9l\" (UniqueName: \"kubernetes.io/projected/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-kube-api-access-2pr9l\") pod \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\" (UID: \"9c1c5977-9c7c-4e88-a11c-eeed34150bbb\") " Oct 29 23:15:19.918336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.918311 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e475c3d4-598c-49dd-afd0-56c090b86f72-kube-api-access-w2wfs\" (UniqueName: \"kubernetes.io/projected/e475c3d4-598c-49dd-afd0-56c090b86f72-kube-api-access-w2wfs\") pod \"e475c3d4-598c-49dd-afd0-56c090b86f72\" (UID: \"e475c3d4-598c-49dd-afd0-56c090b86f72\") " Oct 29 23:15:19.920626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.920548 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e475c3d4-598c-49dd-afd0-56c090b86f72-maas-api-service-ca" pod "e475c3d4-598c-49dd-afd0-56c090b86f72" (UID: "e475c3d4-598c-49dd-afd0-56c090b86f72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:19.920898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.920875 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-maas-api-service-ca" pod "9c1c5977-9c7c-4e88-a11c-eeed34150bbb" (UID: "9c1c5977-9c7c-4e88-a11c-eeed34150bbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:19.927220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.927191 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-kube-api-access-2pr9l" pod "9c1c5977-9c7c-4e88-a11c-eeed34150bbb" (UID: "9c1c5977-9c7c-4e88-a11c-eeed34150bbb"). InnerVolumeSpecName "kube-api-access-2pr9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:19.928778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:19.928755 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e475c3d4-598c-49dd-afd0-56c090b86f72-kube-api-access-w2wfs" pod "e475c3d4-598c-49dd-afd0-56c090b86f72" (UID: "e475c3d4-598c-49dd-afd0-56c090b86f72"). InnerVolumeSpecName "kube-api-access-w2wfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:20.062665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.040524 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:20.066829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.066701 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pr9l\" (UniqueName: \"kubernetes.io/projected/9c1c5977-9c7c-4e88-a11c-eeed34150bbb-kube-api-access-2pr9l\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:20.066829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.066737 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2wfs\" (UniqueName: \"kubernetes.io/projected/e475c3d4-598c-49dd-afd0-56c090b86f72-kube-api-access-w2wfs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:20.066829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.066755 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e475c3d4-598c-49dd-afd0-56c090b86f72-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:20.447480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.447443 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2"] Oct 29 23:15:20.482101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.481800 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" event={"ID":"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3","Type":"ContainerStarted","Data":"3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7"} Oct 29 23:15:20.482101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.481866 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" event={"ID":"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3","Type":"ContainerStarted","Data":"69132e3f676fec56e23f5d47452b67c140c67c888d752f0dfc75f28502e75c76"} Oct 29 23:15:20.486050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.486015 5736 generic.go:356] "Generic (PLEG): container finished" podID="108ecc68-18ab-40e6-a004-6664e74fb746" containerID="1e600d21f4a1f22c1d753a2045d1c5f65490e0e285e7e81bf8f25d336f6cde9d" exitCode=60 Oct 29 23:15:20.486186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.486111 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" event={"ID":"108ecc68-18ab-40e6-a004-6664e74fb746","Type":"ContainerDied","Data":"1e600d21f4a1f22c1d753a2045d1c5f65490e0e285e7e81bf8f25d336f6cde9d"} Oct 29 23:15:20.486186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.486161 5736 scope.go:122] "RemoveContainer" containerID="d447c28d38e837683f9e8062291b159916f10afcbe499e080ff796c7df350b23" Oct 29 23:15:20.486715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.486691 5736 scope.go:122] "RemoveContainer" containerID="1e600d21f4a1f22c1d753a2045d1c5f65490e0e285e7e81bf8f25d336f6cde9d" Oct 29 23:15:20.487080 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:20.487048 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn_odh-ai-gateway-infra(108ecc68-18ab-40e6-a004-6664e74fb746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" Oct 29 23:15:20.494833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.494806 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" Oct 29 23:15:20.494972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.494805 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns" event={"ID":"9c1c5977-9c7c-4e88-a11c-eeed34150bbb","Type":"ContainerDied","Data":"f32e5625cf0cee097aa647e5e95abb54396f0d416595f0392b88eaa9a8865d50"} Oct 29 23:15:20.502295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.502047 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj"] Oct 29 23:15:20.507557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.507532 5736 scope.go:122] "RemoveContainer" containerID="6060f2353ae7b439b548d5a65a07d609803f7dbc9113fa00425033e627f9b00e" Oct 29 23:15:20.507790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.507759 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" Oct 29 23:15:20.507790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.507770 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg" event={"ID":"e475c3d4-598c-49dd-afd0-56c090b86f72","Type":"ContainerDied","Data":"28ebed73669313cc7db09a3cf188c1d5af1d89cd54d4a095faba0c907003f910"} Oct 29 23:15:20.548004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.547529 5736 scope.go:122] "RemoveContainer" containerID="c1114aa163e8dffabdcc68d26426c6a1fedba2ddbcd2c927a4b8c893c1db4bb3" Oct 29 23:15:20.552279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.552083 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" podStartSLOduration=2.552063725 podStartE2EDuration="2.552063725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:20.54041398 +0000 UTC m=+1917.656333601" watchObservedRunningTime="2026-10-29 23:15:20.552063725 +0000 UTC m=+1917.667983331" Oct 29 23:15:20.613970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.613939 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn"] Oct 29 23:15:20.894717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.894671 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp"] Oct 29 23:15:20.919957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.919804 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns"] Oct 29 23:15:20.926611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.926533 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b73a75-29888595-j5mns"] Oct 29 23:15:20.950075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.950043 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg"] Oct 29 23:15:20.962644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.960281 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-96cd49-29888595-k9tdg"] Oct 29 23:15:20.977977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:20.969870 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:21.008742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.008282 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-kube-api-access-5w444\" (UniqueName: \"kubernetes.io/projected/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-kube-api-access-5w444\") pod \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\" (UID: \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\") " Oct 29 23:15:21.008742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.008625 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-maas-api-service-ca\") pod \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\" (UID: \"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3\") " Oct 29 23:15:21.010189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.009822 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-maas-api-service-ca" pod "b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" (UID: "b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:21.019155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.018724 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.047936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.047041 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-kube-api-access-5w444" pod "b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" (UID: "b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3"). InnerVolumeSpecName "kube-api-access-5w444". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:21.057146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.057107 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" path="/var/lib/kubelet/pods/2ec07a79-6535-4a59-abef-3e1b93df3a3c/volumes" Oct 29 23:15:21.058747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.058083 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" path="/var/lib/kubelet/pods/8b89fbf3-5d28-4308-80fc-d7d51192e711/volumes" Oct 29 23:15:21.060087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.059615 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" path="/var/lib/kubelet/pods/9c1c5977-9c7c-4e88-a11c-eeed34150bbb/volumes" Oct 29 23:15:21.060717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.060694 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" path="/var/lib/kubelet/pods/e475c3d4-598c-49dd-afd0-56c090b86f72/volumes" Oct 29 23:15:21.188018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.135094 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w444\" (UniqueName: \"kubernetes.io/projected/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3-kube-api-access-5w444\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.188018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.148885 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:21.322992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.306443 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:21.322992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.314734 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:21.325080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.324958 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e497886-fd10-4e77-8d58-8ba13a9b3c89-kube-api-access-4j4z8\" (UniqueName: \"kubernetes.io/projected/6e497886-fd10-4e77-8d58-8ba13a9b3c89-kube-api-access-4j4z8\") pod \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\" (UID: \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\") " Oct 29 23:15:21.349610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.349536 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e497886-fd10-4e77-8d58-8ba13a9b3c89-kube-api-access-4j4z8" pod "6e497886-fd10-4e77-8d58-8ba13a9b3c89" (UID: "6e497886-fd10-4e77-8d58-8ba13a9b3c89"). InnerVolumeSpecName "kube-api-access-4j4z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:21.442399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.442095 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e497886-fd10-4e77-8d58-8ba13a9b3c89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e497886-fd10-4e77-8d58-8ba13a9b3c89-maas-api-service-ca\") pod \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\" (UID: \"6e497886-fd10-4e77-8d58-8ba13a9b3c89\") " Oct 29 23:15:21.443167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.443145 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dab4a21-963e-4dde-9bb1-835f436ad9a9-kube-api-access-7gzbv\" (UniqueName: \"kubernetes.io/projected/9dab4a21-963e-4dde-9bb1-835f436ad9a9-kube-api-access-7gzbv\") pod \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\" (UID: \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\") " Oct 29 23:15:21.443534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.443070 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e497886-fd10-4e77-8d58-8ba13a9b3c89-maas-api-service-ca" pod "6e497886-fd10-4e77-8d58-8ba13a9b3c89" (UID: "6e497886-fd10-4e77-8d58-8ba13a9b3c89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:21.444099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.444081 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dab4a21-963e-4dde-9bb1-835f436ad9a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dab4a21-963e-4dde-9bb1-835f436ad9a9-maas-api-service-ca\") pod \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\" (UID: \"9dab4a21-963e-4dde-9bb1-835f436ad9a9\") " Oct 29 23:15:21.444935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.444914 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j4z8\" (UniqueName: \"kubernetes.io/projected/6e497886-fd10-4e77-8d58-8ba13a9b3c89-kube-api-access-4j4z8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.445069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.445056 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e497886-fd10-4e77-8d58-8ba13a9b3c89-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.445518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.445498 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dab4a21-963e-4dde-9bb1-835f436ad9a9-maas-api-service-ca" pod "9dab4a21-963e-4dde-9bb1-835f436ad9a9" (UID: "9dab4a21-963e-4dde-9bb1-835f436ad9a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:21.463248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.463219 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dab4a21-963e-4dde-9bb1-835f436ad9a9-kube-api-access-7gzbv" pod "9dab4a21-963e-4dde-9bb1-835f436ad9a9" (UID: "9dab4a21-963e-4dde-9bb1-835f436ad9a9"). InnerVolumeSpecName "kube-api-access-7gzbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:21.494930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.494887 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:21.553776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.552360 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a141e7-617e-4172-9c0c-d19df1046266-kube-api-access-b8w5f\" (UniqueName: \"kubernetes.io/projected/83a141e7-617e-4172-9c0c-d19df1046266-kube-api-access-b8w5f\") pod \"83a141e7-617e-4172-9c0c-d19df1046266\" (UID: \"83a141e7-617e-4172-9c0c-d19df1046266\") " Oct 29 23:15:21.553776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.552688 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a141e7-617e-4172-9c0c-d19df1046266-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a141e7-617e-4172-9c0c-d19df1046266-maas-api-service-ca\") pod \"83a141e7-617e-4172-9c0c-d19df1046266\" (UID: \"83a141e7-617e-4172-9c0c-d19df1046266\") " Oct 29 23:15:21.554810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.554511 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzbv\" (UniqueName: \"kubernetes.io/projected/9dab4a21-963e-4dde-9bb1-835f436ad9a9-kube-api-access-7gzbv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.554810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.554545 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dab4a21-963e-4dde-9bb1-835f436ad9a9-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.570600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.560815 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a141e7-617e-4172-9c0c-d19df1046266-kube-api-access-b8w5f" pod "83a141e7-617e-4172-9c0c-d19df1046266" (UID: "83a141e7-617e-4172-9c0c-d19df1046266"). InnerVolumeSpecName "kube-api-access-b8w5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:21.570600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.563905 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a141e7-617e-4172-9c0c-d19df1046266-maas-api-service-ca" pod "83a141e7-617e-4172-9c0c-d19df1046266" (UID: "83a141e7-617e-4172-9c0c-d19df1046266"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:21.614573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.613398 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" event={"ID":"6e497886-fd10-4e77-8d58-8ba13a9b3c89","Type":"ContainerDied","Data":"6cb674b913b7681f088b34a47d79ff7a1dc11a407dfe6d4ea2de576b7489b3b9"} Oct 29 23:15:21.614573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.613496 5736 scope.go:122] "RemoveContainer" containerID="62ccd5db1710cb72644c1f495361805992353494f88e39c9a4019b65dac0a63b" Oct 29 23:15:21.614573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.613651 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6" Oct 29 23:15:21.624521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.624209 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" Oct 29 23:15:21.624521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.624227 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2" event={"ID":"9dab4a21-963e-4dde-9bb1-835f436ad9a9","Type":"ContainerDied","Data":"6c3dce155dddabce7723299965a828c7a930abd7c65b91ccd4232f14bf2a1375"} Oct 29 23:15:21.645573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.645536 5736 scope.go:122] "RemoveContainer" containerID="df14cba9107b57a871c00f0cf8c7198a5cd6e95eaaf0b6d10b37713e941ae757" Oct 29 23:15:21.646075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.645691 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" Oct 29 23:15:21.646075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.645937 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj" event={"ID":"b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3","Type":"ContainerDied","Data":"e148dd59bc45c1daa5a680e7249b9a38bfa530450cde582797831ab720f6887d"} Oct 29 23:15:21.650309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.650229 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" Oct 29 23:15:21.650309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.650262 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp" event={"ID":"83a141e7-617e-4172-9c0c-d19df1046266","Type":"ContainerDied","Data":"0534b60e76a1e41b33978e7bdd78b1544cce566d9a37a16f8bad5cc7c53cd0f0"} Oct 29 23:15:21.656870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.656077 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8w5f\" (UniqueName: \"kubernetes.io/projected/83a141e7-617e-4172-9c0c-d19df1046266-kube-api-access-b8w5f\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.656870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.656105 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a141e7-617e-4172-9c0c-d19df1046266-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.666169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.665962 5736 scope.go:122] "RemoveContainer" containerID="d78669ddbba7ced3b4a43d5348ee0bb717471b69d67b8c6b05feb3caf8cbbdcc" Oct 29 23:15:21.685952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.685439 5736 scope.go:122] "RemoveContainer" containerID="f33f64793fb8eaedc08e2787e7f0b34a4ab595ac483b1f97cf11fd218975c5fc" Oct 29 23:15:21.866602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.859800 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj"] Oct 29 23:15:21.866602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.859858 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888595-nlrmj"] Oct 29 23:15:21.868418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.866877 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2"] Oct 29 23:15:21.879088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.879064 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c83b50-29888595-zdcf2"] Oct 29 23:15:21.884557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.884533 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:21.888049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.887999 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp"] Oct 29 23:15:21.895178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.895151 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888595-h25lp"] Oct 29 23:15:21.906052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.906022 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6"] Oct 29 23:15:21.912552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:21.912528 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888595-77hh6"] Oct 29 23:15:22.020435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.020398 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108ecc68-18ab-40e6-a004-6664e74fb746-kube-api-access-d879n\" (UniqueName: \"kubernetes.io/projected/108ecc68-18ab-40e6-a004-6664e74fb746-kube-api-access-d879n\") pod \"108ecc68-18ab-40e6-a004-6664e74fb746\" (UID: \"108ecc68-18ab-40e6-a004-6664e74fb746\") " Oct 29 23:15:22.020640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.020539 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108ecc68-18ab-40e6-a004-6664e74fb746-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108ecc68-18ab-40e6-a004-6664e74fb746-maas-api-service-ca\") pod \"108ecc68-18ab-40e6-a004-6664e74fb746\" (UID: \"108ecc68-18ab-40e6-a004-6664e74fb746\") " Oct 29 23:15:22.020939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.020898 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108ecc68-18ab-40e6-a004-6664e74fb746-maas-api-service-ca" pod "108ecc68-18ab-40e6-a004-6664e74fb746" (UID: "108ecc68-18ab-40e6-a004-6664e74fb746"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:22.021264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.021244 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108ecc68-18ab-40e6-a004-6664e74fb746-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.022723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.022697 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108ecc68-18ab-40e6-a004-6664e74fb746-kube-api-access-d879n" pod "108ecc68-18ab-40e6-a004-6664e74fb746" (UID: "108ecc68-18ab-40e6-a004-6664e74fb746"). InnerVolumeSpecName "kube-api-access-d879n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:22.123001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.122963 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d879n\" (UniqueName: \"kubernetes.io/projected/108ecc68-18ab-40e6-a004-6664e74fb746-kube-api-access-d879n\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.667639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.667278 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" Oct 29 23:15:22.668179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.667283 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn" event={"ID":"108ecc68-18ab-40e6-a004-6664e74fb746","Type":"ContainerDied","Data":"b29d6bd09cf9d96cc0927c8e2a80a67326458fd1fc1d7ed96f6a06e9a4681519"} Oct 29 23:15:22.668281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.668226 5736 scope.go:122] "RemoveContainer" containerID="1e600d21f4a1f22c1d753a2045d1c5f65490e0e285e7e81bf8f25d336f6cde9d" Oct 29 23:15:22.726319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.726283 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn"] Oct 29 23:15:22.730918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:22.730891 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888595-ms4dn"] Oct 29 23:15:23.009067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:23.009027 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" path="/var/lib/kubelet/pods/108ecc68-18ab-40e6-a004-6664e74fb746/volumes" Oct 29 23:15:23.009806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:23.009784 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" path="/var/lib/kubelet/pods/6e497886-fd10-4e77-8d58-8ba13a9b3c89/volumes" Oct 29 23:15:23.010420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:23.010402 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a141e7-617e-4172-9c0c-d19df1046266" path="/var/lib/kubelet/pods/83a141e7-617e-4172-9c0c-d19df1046266/volumes" Oct 29 23:15:23.011512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:23.011493 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" path="/var/lib/kubelet/pods/9dab4a21-963e-4dde-9bb1-835f436ad9a9/volumes" Oct 29 23:15:23.012156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:23.012137 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" path="/var/lib/kubelet/pods/b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3/volumes" Oct 29 23:15:24.139418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.139344 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm"] Oct 29 23:15:24.140081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140064 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerName="cleanup" Oct 29 23:15:24.140081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140082 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140093 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140099 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140109 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140115 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140122 5736 cpu_manager.go:414] "containerMap: removing container" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140126 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140137 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140141 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140151 5736 cpu_manager.go:414] "containerMap: removing container" podUID="83a141e7-617e-4172-9c0c-d19df1046266" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140156 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a141e7-617e-4172-9c0c-d19df1046266" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140175 5736 cpu_manager.go:414] "containerMap: removing container" podUID="83a141e7-617e-4172-9c0c-d19df1046266" containerName="cleanup" Oct 29 23:15:24.140180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140180 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a141e7-617e-4172-9c0c-d19df1046266" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140187 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140192 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140198 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140202 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140210 5736 cpu_manager.go:414] "containerMap: removing container" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140214 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140227 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140231 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140238 5736 cpu_manager.go:414] "containerMap: removing container" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140242 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140261 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140266 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140274 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140279 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140286 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140290 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140297 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140301 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140308 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140313 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140319 5736 cpu_manager.go:414] "containerMap: removing container" podUID="83a141e7-617e-4172-9c0c-d19df1046266" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140324 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a141e7-617e-4172-9c0c-d19df1046266" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140420 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a141e7-617e-4172-9c0c-d19df1046266" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140427 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140434 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b89fbf3-5d28-4308-80fc-d7d51192e711" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140441 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140448 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140454 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec07a79-6535-4a59-abef-3e1b93df3a3c" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140460 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140466 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140472 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140477 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140483 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140487 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140493 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140498 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140504 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140510 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140514 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140519 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="108ecc68-18ab-40e6-a004-6664e74fb746" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140525 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerName="cleanup" Oct 29 23:15:24.140567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.140529 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a141e7-617e-4172-9c0c-d19df1046266" containerName="cleanup" Oct 29 23:15:24.143307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.143289 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.166864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.166823 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm"] Oct 29 23:15:24.172261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.172198 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-26a2a4\"/\"openshift-service-ca.crt\"" Oct 29 23:15:24.172412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.172291 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-26a2a4\"/\"kube-root-ca.crt\"" Oct 29 23:15:24.175817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.175791 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-26a2a4\"/\"auth-test-model-26a2a4-kserve-self-signed-certs\"" Oct 29 23:15:24.175817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.175793 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-26a2a4\"/\"default-dockercfg-h9tg2\"" Oct 29 23:15:24.260357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.260317 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9dfbdd-401e-4f49-bb40-176392c82957-tls-certs\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.260553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.260392 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-model-cache\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.260553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.260509 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-tmp-dir\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.260698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.260661 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njkqn\" (UniqueName: \"kubernetes.io/projected/1c9dfbdd-401e-4f49-bb40-176392c82957-kube-api-access-njkqn\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.260856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.260840 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-dshm\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.260962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.260947 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-home\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.362381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.362340 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njkqn\" (UniqueName: \"kubernetes.io/projected/1c9dfbdd-401e-4f49-bb40-176392c82957-kube-api-access-njkqn\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.362562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.362527 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-dshm\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.362671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.362641 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-home\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.362763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.362747 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9dfbdd-401e-4f49-bb40-176392c82957-tls-certs\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.362811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.362797 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-model-cache\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.363218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.362881 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-tmp-dir\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.363218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.362996 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-home\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.363218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.363172 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-model-cache\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.363409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.363268 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-tmp-dir\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.365519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.365497 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-dshm\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.365782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.365751 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9dfbdd-401e-4f49-bb40-176392c82957-tls-certs\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.394370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.394336 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njkqn\" (UniqueName: \"kubernetes.io/projected/1c9dfbdd-401e-4f49-bb40-176392c82957-kube-api-access-njkqn\") pod \"auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.468652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.461872 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:24.713045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.713011 5736 generic.go:356] "Generic (PLEG): container finished" podID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerID="5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955" exitCode=0 Oct 29 23:15:24.713409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.713381 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" event={"ID":"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4","Type":"ContainerDied","Data":"5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955"} Oct 29 23:15:24.714066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.714043 5736 scope.go:122] "RemoveContainer" containerID="5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955" Oct 29 23:15:24.717707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.717677 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:15:24.717815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.717764 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:15:24.717884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.717814 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:15:24.937174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:24.937140 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm"] Oct 29 23:15:24.938192 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:24.938168 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c9dfbdd_401e_4f49_bb40_176392c82957.slice/crio-d19a97127f77f839000500f973c5224e9650776f4aeb4b60ef63264ccc348ce6 WatchSource:0}: Error finding container d19a97127f77f839000500f973c5224e9650776f4aeb4b60ef63264ccc348ce6: Status 404 returned error can't find the container with id d19a97127f77f839000500f973c5224e9650776f4aeb4b60ef63264ccc348ce6 Oct 29 23:15:25.270517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:25.270439 5736 scope.go:122] "RemoveContainer" containerID="30dfde2f9363822e5e47cce685506cb92fee8755173c6dae0290226d065c4c87" Oct 29 23:15:25.287597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:25.283749 5736 scope.go:122] "RemoveContainer" containerID="40ee5546b6aaf1347b61ad309a9aaedb6deb05ffef8c93aa44aac5c423504bd6" Oct 29 23:15:25.307602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:25.298628 5736 scope.go:122] "RemoveContainer" containerID="731728df20f6118d7eb5f53f3d11e03ecac8a16d05040e09753ddc29c037b99c" Oct 29 23:15:25.754617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:25.753335 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" event={"ID":"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4","Type":"ContainerStarted","Data":"6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8"} Oct 29 23:15:25.757273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:25.757233 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" event={"ID":"1c9dfbdd-401e-4f49-bb40-176392c82957","Type":"ContainerStarted","Data":"1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d"} Oct 29 23:15:25.758422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:25.758401 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" event={"ID":"1c9dfbdd-401e-4f49-bb40-176392c82957","Type":"ContainerStarted","Data":"d19a97127f77f839000500f973c5224e9650776f4aeb4b60ef63264ccc348ce6"} Oct 29 23:15:25.893158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:25.893046 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" podStartSLOduration=1.893030363 podStartE2EDuration="1.893030363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:25.888840707 +0000 UTC m=+1923.004760310" watchObservedRunningTime="2026-10-29 23:15:25.893030363 +0000 UTC m=+1923.008949970" Oct 29 23:15:26.276107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:26.276073 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:15:26.277296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:26.277266 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:26.277411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:26.277319 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 29 23:15:26.462945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:26.462899 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:26.826393 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:26.826351 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:15:27.276770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:27.276735 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:27.277276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:27.276792 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 29 23:15:28.277088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.277052 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:28.277504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.277105 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 29 23:15:28.664043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.663957 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c"] Oct 29 23:15:28.664811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.664788 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerName="cleanup" Oct 29 23:15:28.664811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.664812 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e497886-fd10-4e77-8d58-8ba13a9b3c89" containerName="cleanup" Oct 29 23:15:28.665000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.664842 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerName="cleanup" Oct 29 23:15:28.665000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.664863 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16afaed-6eda-4bc6-86dc-d92e6b8c2ea3" containerName="cleanup" Oct 29 23:15:28.665000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.664888 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerName="cleanup" Oct 29 23:15:28.665000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.664897 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e475c3d4-598c-49dd-afd0-56c090b86f72" containerName="cleanup" Oct 29 23:15:28.665000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.664933 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerName="cleanup" Oct 29 23:15:28.665000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.664940 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerName="cleanup" Oct 29 23:15:28.665318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.665136 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a141e7-617e-4172-9c0c-d19df1046266" containerName="cleanup" Oct 29 23:15:28.665318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.665154 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1c5977-9c7c-4e88-a11c-eeed34150bbb" containerName="cleanup" Oct 29 23:15:28.665318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.665172 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dab4a21-963e-4dde-9bb1-835f436ad9a9" containerName="cleanup" Oct 29 23:15:28.673602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.669652 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.680805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.680771 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a216c693-7350-49a8-b979-a706ae70da07-tls-certs\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.680948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.680825 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-dshm\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.681820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.681789 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-tmp-dir\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.681941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.681873 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-model-cache\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.681941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.681915 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8r5\" (UniqueName: \"kubernetes.io/projected/a216c693-7350-49a8-b979-a706ae70da07-kube-api-access-bx8r5\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.682062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.681992 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-home\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.695105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.695081 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c"] Oct 29 23:15:28.696279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.696252 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-b4b5e3\"/\"kube-root-ca.crt\"" Oct 29 23:15:28.699122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.699084 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-b4b5e3\"/\"default-dockercfg-44gv8\"" Oct 29 23:15:28.699261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.699144 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-b4b5e3\"/\"auth-test-model-b4b5e3-kserve-self-signed-certs\"" Oct 29 23:15:28.699261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.699216 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-b4b5e3\"/\"openshift-service-ca.crt\"" Oct 29 23:15:28.784164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.784118 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a216c693-7350-49a8-b979-a706ae70da07-tls-certs\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.784377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.784174 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-dshm\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.784377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.784317 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-tmp-dir\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.784517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.784388 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-model-cache\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.784517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.784431 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8r5\" (UniqueName: \"kubernetes.io/projected/a216c693-7350-49a8-b979-a706ae70da07-kube-api-access-bx8r5\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.784517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.784509 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-home\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.785165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.785114 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-home\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.787218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.787164 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-model-cache\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.787839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.787815 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-tmp-dir\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.790732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.790712 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a216c693-7350-49a8-b979-a706ae70da07-tls-certs\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.791760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.791737 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-dshm\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.810703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.810675 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8r5\" (UniqueName: \"kubernetes.io/projected/a216c693-7350-49a8-b979-a706ae70da07-kube-api-access-bx8r5\") pod \"auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:28.996857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:28.996817 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:29.276232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:29.276136 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:29.279265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:29.279223 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 29 23:15:29.284159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:29.284137 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:29.290196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:29.290172 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:29.706045 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:29.705898 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda216c693_7350_49a8_b979_a706ae70da07.slice/crio-0ebebf9413ad7da76263b46fb9fd85eb6a78441790c68ece76ab2cf9aa3c1775 WatchSource:0}: Error finding container 0ebebf9413ad7da76263b46fb9fd85eb6a78441790c68ece76ab2cf9aa3c1775: Status 404 returned error can't find the container with id 0ebebf9413ad7da76263b46fb9fd85eb6a78441790c68ece76ab2cf9aa3c1775 Oct 29 23:15:29.724462 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:29.724426 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c"] Oct 29 23:15:29.811967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:29.811931 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" event={"ID":"a216c693-7350-49a8-b979-a706ae70da07","Type":"ContainerStarted","Data":"0ebebf9413ad7da76263b46fb9fd85eb6a78441790c68ece76ab2cf9aa3c1775"} Oct 29 23:15:29.823819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:29.823795 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:15:30.276770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:30.276738 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 29 23:15:30.276948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:30.276800 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 29 23:15:30.282886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:30.282853 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:15:30.823358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:30.823323 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" event={"ID":"a216c693-7350-49a8-b979-a706ae70da07","Type":"ContainerStarted","Data":"fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c"} Oct 29 23:15:30.872662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:30.872600 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" podStartSLOduration=2.872569804 podStartE2EDuration="2.872569804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:30.872030649 +0000 UTC m=+1927.987950267" watchObservedRunningTime="2026-10-29 23:15:30.872569804 +0000 UTC m=+1927.988489411" Oct 29 23:15:31.002709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:30.999678 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:31.281156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:31.281119 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:15:31.282279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:31.282254 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:15:34.462786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:34.462748 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:34.476060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:34.476034 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:34.868214 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:34.868127 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:15:39.007096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:39.007057 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:39.011537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:39.011508 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:39.044704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:39.044667 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq"] Oct 29 23:15:39.049041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:39.049012 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c748663d-s9msq"] Oct 29 23:15:39.929001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:39.928970 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:15:40.033139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:40.033085 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5"] Oct 29 23:15:40.048661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:40.048611 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57df55e1-cnmz5"] Oct 29 23:15:40.054480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:40.054450 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c"] Oct 29 23:15:40.058407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:40.058385 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29365cb-q9t8c"] Oct 29 23:15:41.006388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:41.006357 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec983e3-0680-4148-ba8e-393cb06599f9" path="/var/lib/kubelet/pods/1ec983e3-0680-4148-ba8e-393cb06599f9/volumes" Oct 29 23:15:41.006902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:41.006886 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b418bcd-7812-4d03-9d5e-3581982db4d2" path="/var/lib/kubelet/pods/2b418bcd-7812-4d03-9d5e-3581982db4d2/volumes" Oct 29 23:15:41.007331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:41.007317 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912c4231-8553-491e-8716-536551e89ded" path="/var/lib/kubelet/pods/912c4231-8553-491e-8716-536551e89ded/volumes" Oct 29 23:15:45.628794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.628746 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb"] Oct 29 23:15:45.658387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.658355 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.672411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.672382 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb"] Oct 29 23:15:45.678717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.678692 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-96cd49-openshift-default-dockercfg-959wn\"" Oct 29 23:15:45.727056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.727022 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde8163b-c690-4511-ad4d-ba7acee99494-istio-podinfo\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.727225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.727098 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-credential-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.727225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.727192 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde8163b-c690-4511-ad4d-ba7acee99494-istiod-ca-cert\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.727317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.727280 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-certs\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.727450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.727428 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-data\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.727643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.727622 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-istio-token\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.727809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.727785 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g64j\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-kube-api-access-6g64j\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.727937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.727826 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-envoy\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.727937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.727877 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.829731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.829698 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-data\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.829892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.829861 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-istio-token\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.829892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.829882 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g64j\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-kube-api-access-6g64j\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.829968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.829907 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-envoy\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.829968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.829955 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.830064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.830041 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde8163b-c690-4511-ad4d-ba7acee99494-istio-podinfo\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.830123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.830099 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-data\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.830180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.830103 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-credential-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.830252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.830178 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde8163b-c690-4511-ad4d-ba7acee99494-istiod-ca-cert\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.830322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.830254 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-certs\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.830420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.830387 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.853602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.830549 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-credential-socket\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.853602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.830997 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde8163b-c690-4511-ad4d-ba7acee99494-istiod-ca-cert\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.857906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.857562 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g64j\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-kube-api-access-6g64j\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.858928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.858898 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde8163b-c690-4511-ad4d-ba7acee99494-istio-podinfo\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.859161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.859138 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-envoy\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.860852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.860827 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-certs\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.861269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.861240 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-istio-token\") pod \"e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:45.975062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:45.975026 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:46.325198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:46.325165 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb"] Oct 29 23:15:46.325970 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:15:46.325943 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde8163b_c690_4511_ad4d_ba7acee99494.slice/crio-a577ef9614d5446112bf094b9aae6aa1b6e14ee30bc2e0ffb70540751bb55c63 WatchSource:0}: Error finding container a577ef9614d5446112bf094b9aae6aa1b6e14ee30bc2e0ffb70540751bb55c63: Status 404 returned error can't find the container with id a577ef9614d5446112bf094b9aae6aa1b6e14ee30bc2e0ffb70540751bb55c63 Oct 29 23:15:46.329435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:46.329403 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:15:46.329511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:46.329469 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:15:46.329511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:46.329500 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:15:46.591975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:46.591894 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4"] Oct 29 23:15:47.061601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.058758 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc"] Oct 29 23:15:47.107667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.106147 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.109603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.108143 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc"] Oct 29 23:15:47.124791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.124765 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:15:47.125003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.124982 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:15:47.167781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.157486 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" event={"ID":"dde8163b-c690-4511-ad4d-ba7acee99494","Type":"ContainerStarted","Data":"c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab"} Oct 29 23:15:47.167781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.157591 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" event={"ID":"dde8163b-c690-4511-ad4d-ba7acee99494","Type":"ContainerStarted","Data":"a577ef9614d5446112bf094b9aae6aa1b6e14ee30bc2e0ffb70540751bb55c63"} Oct 29 23:15:47.219353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.219316 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2rc2\" (UniqueName: \"kubernetes.io/projected/6851d12c-f899-4e6f-b024-99f2d1d9b449-kube-api-access-m2rc2\") pod \"maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.219571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.219490 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.219681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.219608 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.282107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.281812 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" podStartSLOduration=2.281792125 podStartE2EDuration="2.281792125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:47.278970906 +0000 UTC m=+1944.394890518" watchObservedRunningTime="2026-10-29 23:15:47.281792125 +0000 UTC m=+1944.397711732" Oct 29 23:15:47.321140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.321043 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.321313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.321173 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.321569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.321545 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2rc2\" (UniqueName: \"kubernetes.io/projected/6851d12c-f899-4e6f-b024-99f2d1d9b449-kube-api-access-m2rc2\") pod \"maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.321952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.321917 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.325339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.325311 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.356599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.356381 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2rc2\" (UniqueName: \"kubernetes.io/projected/6851d12c-f899-4e6f-b024-99f2d1d9b449-kube-api-access-m2rc2\") pod \"maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.489752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.489715 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:47.834609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.831606 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc"] Oct 29 23:15:47.976167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.976129 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:47.978009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.977977 5736 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:15:47.978190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:47.978032 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:15:48.167479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:48.167446 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" event={"ID":"6851d12c-f899-4e6f-b024-99f2d1d9b449","Type":"ContainerStarted","Data":"1911547e164bc4e751b5ecdb371f7476a4ccd88753c4981b7471c5bfec33cd5b"} Oct 29 23:15:48.975393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:48.975353 5736 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:15:48.975572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:48.975420 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:15:49.178646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:49.178601 5736 generic.go:356] "Generic (PLEG): container finished" podID="6851d12c-f899-4e6f-b024-99f2d1d9b449" containerID="8206797de78d7b2eaa7eae8a4b340fa1114c0fd8e7c287a73d24f9aefb6a1518" exitCode=0 Oct 29 23:15:49.179968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:49.179937 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" event={"ID":"6851d12c-f899-4e6f-b024-99f2d1d9b449","Type":"ContainerDied","Data":"8206797de78d7b2eaa7eae8a4b340fa1114c0fd8e7c287a73d24f9aefb6a1518"} Oct 29 23:15:49.976154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:49.976119 5736 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:15:49.976470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:49.976171 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:15:50.035613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.034651 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4"] Oct 29 23:15:50.066780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.066738 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745c9def-wk8z4"] Oct 29 23:15:50.474193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.474163 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:50.514335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.514302 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-openshift-ca\") pod \"6851d12c-f899-4e6f-b024-99f2d1d9b449\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " Oct 29 23:15:50.514507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.514493 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-xks-ca\") pod \"6851d12c-f899-4e6f-b024-99f2d1d9b449\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " Oct 29 23:15:50.514633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.514612 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6851d12c-f899-4e6f-b024-99f2d1d9b449-kube-api-access-m2rc2\" (UniqueName: \"kubernetes.io/projected/6851d12c-f899-4e6f-b024-99f2d1d9b449-kube-api-access-m2rc2\") pod \"6851d12c-f899-4e6f-b024-99f2d1d9b449\" (UID: \"6851d12c-f899-4e6f-b024-99f2d1d9b449\") " Oct 29 23:15:50.514741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.514661 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-openshift-ca" pod "6851d12c-f899-4e6f-b024-99f2d1d9b449" (UID: "6851d12c-f899-4e6f-b024-99f2d1d9b449"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:50.515624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.515548 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:50.518312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.518076 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6851d12c-f899-4e6f-b024-99f2d1d9b449-kube-api-access-m2rc2" pod "6851d12c-f899-4e6f-b024-99f2d1d9b449" (UID: "6851d12c-f899-4e6f-b024-99f2d1d9b449"). InnerVolumeSpecName "kube-api-access-m2rc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:50.518758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.518721 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-xks-ca" pod "6851d12c-f899-4e6f-b024-99f2d1d9b449" (UID: "6851d12c-f899-4e6f-b024-99f2d1d9b449"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:50.617209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.617175 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6851d12c-f899-4e6f-b024-99f2d1d9b449-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:50.617209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.617203 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2rc2\" (UniqueName: \"kubernetes.io/projected/6851d12c-f899-4e6f-b024-99f2d1d9b449-kube-api-access-m2rc2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:15:50.918737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.918609 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj"] Oct 29 23:15:50.919503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.919476 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6851d12c-f899-4e6f-b024-99f2d1d9b449" containerName="revoke-keys" Oct 29 23:15:50.919503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.919499 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6851d12c-f899-4e6f-b024-99f2d1d9b449" containerName="revoke-keys" Oct 29 23:15:50.919743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.919656 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6851d12c-f899-4e6f-b024-99f2d1d9b449" containerName="revoke-keys" Oct 29 23:15:50.923948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.923923 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:50.965604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.963257 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2126fb\"/\"kube-root-ca.crt\"" Oct 29 23:15:50.965604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.963343 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2126fb\"/\"openshift-service-ca.crt\"" Oct 29 23:15:50.976159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.976126 5736 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:15:50.976306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.976188 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:15:50.982983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:50.978868 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj"] Oct 29 23:15:51.010411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.009949 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b71ee4-4a70-49d7-960d-54d8ab1a2360" path="/var/lib/kubelet/pods/93b71ee4-4a70-49d7-960d-54d8ab1a2360/volumes" Oct 29 23:15:51.021612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.021546 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2126fb\"/\"auth-test-model-2126fb-kserve-self-signed-certs\"" Oct 29 23:15:51.021612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.021574 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2126fb\"/\"default-dockercfg-f88fn\"" Oct 29 23:15:51.025480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.025452 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-model-cache\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.025651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.025629 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tls-certs\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.025882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.025863 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78sxt\" (UniqueName: \"kubernetes.io/projected/87e5da3a-f63d-4133-a5cb-90bc11aaa123-kube-api-access-78sxt\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.025961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.025949 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-home\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.026023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.025968 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tmp-dir\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.026023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.025985 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-dshm\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.073693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.073642 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps"] Oct 29 23:15:51.078298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.078213 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29072a62-ltsps"] Oct 29 23:15:51.127974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.127940 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78sxt\" (UniqueName: \"kubernetes.io/projected/87e5da3a-f63d-4133-a5cb-90bc11aaa123-kube-api-access-78sxt\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.128121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.128052 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-home\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.128121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.128084 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tmp-dir\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.128121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.128108 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-dshm\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.128246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.128214 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-model-cache\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.128322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.128306 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tls-certs\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.128590 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.128550 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tmp-dir\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.128877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.128854 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-model-cache\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.128968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.128921 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-home\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.135608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.131257 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-dshm\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.147605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.146153 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tls-certs\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.158312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.158255 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78sxt\" (UniqueName: \"kubernetes.io/projected/87e5da3a-f63d-4133-a5cb-90bc11aaa123-kube-api-access-78sxt\") pod \"auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.289628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.289565 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" event={"ID":"6851d12c-f899-4e6f-b024-99f2d1d9b449","Type":"ContainerDied","Data":"1911547e164bc4e751b5ecdb371f7476a4ccd88753c4981b7471c5bfec33cd5b"} Oct 29 23:15:51.291040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.291000 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1911547e164bc4e751b5ecdb371f7476a4ccd88753c4981b7471c5bfec33cd5b" Oct 29 23:15:51.291225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.290287 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:51.291557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.290023 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc" Oct 29 23:15:51.869936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.869843 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj"] Oct 29 23:15:51.982242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.982207 5736 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:15:51.982383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:51.982265 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:15:52.300223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:52.300185 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" event={"ID":"87e5da3a-f63d-4133-a5cb-90bc11aaa123","Type":"ContainerStarted","Data":"de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed"} Oct 29 23:15:52.300223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:52.300237 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" event={"ID":"87e5da3a-f63d-4133-a5cb-90bc11aaa123","Type":"ContainerStarted","Data":"a5b9ca71581681e5b50ac36bfa48b717b92d5105f1738f9dde2a4ac894d86194"} Oct 29 23:15:52.975387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:52.975354 5736 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:15:52.975923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:52.975414 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:15:53.006786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:53.006750 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af3095e-02e5-4b19-9a36-22d595f9b64f" path="/var/lib/kubelet/pods/6af3095e-02e5-4b19-9a36-22d595f9b64f/volumes" Oct 29 23:15:53.291480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:53.291398 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:15:53.773271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:53.773238 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": read tcp 10.134.0.2:43860->10.134.0.222:15021: read: connection reset by peer" start-of-body= Oct 29 23:15:53.773449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:53.773306 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": read tcp 10.134.0.2:43860->10.134.0.222:15021: read: connection reset by peer" Oct 29 23:15:53.975831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:53.975786 5736 patch_prober.go:29] interesting pod/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:15:53.976213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:53.975855 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:15:54.031572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.031456 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" podStartSLOduration=4.031437478 podStartE2EDuration="4.031437478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:52.358987582 +0000 UTC m=+1949.474907214" watchObservedRunningTime="2026-10-29 23:15:54.031437478 +0000 UTC m=+1951.147357085" Oct 29 23:15:54.034118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.034088 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg"] Oct 29 23:15:54.041048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.041022 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2"] Oct 29 23:15:54.045154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.045128 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a29c5bfc-mz7zg"] Oct 29 23:15:54.048732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.048709 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b239048-l6sj2"] Oct 29 23:15:54.322427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.322346 5736 generic.go:356] "Generic (PLEG): container finished" podID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerID="27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61" exitCode=0 Oct 29 23:15:54.322655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.322421 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" event={"ID":"b002e9fa-a1b2-4a58-9468-0a6de1a35017","Type":"ContainerDied","Data":"27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61"} Oct 29 23:15:54.323081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.323060 5736 scope.go:122] "RemoveContainer" containerID="27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61" Oct 29 23:15:54.325046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.325020 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:15:54.325142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.325107 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:15:54.325184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.325154 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:15:54.980957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.980931 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:54.981376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.981358 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:54.982107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:54.982090 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:15:55.006307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:55.006282 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363e8920-b210-4a9c-86fb-c9049b44c0e6" path="/var/lib/kubelet/pods/363e8920-b210-4a9c-86fb-c9049b44c0e6/volumes" Oct 29 23:15:55.006818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:55.006801 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e80024-ec48-4ecf-bfab-6f6898768b5d" path="/var/lib/kubelet/pods/c1e80024-ec48-4ecf-bfab-6f6898768b5d/volumes" Oct 29 23:15:55.349658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:55.337426 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" event={"ID":"b002e9fa-a1b2-4a58-9468-0a6de1a35017","Type":"ContainerStarted","Data":"2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9"} Oct 29 23:15:56.257639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:56.255761 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:15:56.257639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:56.256192 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:56.257639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:56.256241 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 29 23:15:56.826194 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:15:56.826146 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:15:57.250600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:57.248029 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:57.250600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:57.248092 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 29 23:15:58.252650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:58.238054 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:58.252650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:58.238115 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 29 23:15:59.258602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:59.257760 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:15:59.258602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:15:59.257824 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 29 23:16:00.250605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:00.242981 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 29 23:16:00.250605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:00.243045 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 29 23:16:01.242570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.242535 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:16:01.243009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.242990 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:16:01.243656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.243629 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:16:01.303607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.295847 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:16:01.334111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.334079 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:16:01.512774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.512688 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:16:01.712464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.712427 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" podUID="d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" containerName="main" containerID="cri-o://3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7" gracePeriod=45 Oct 29 23:16:01.812434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.812359 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" podUID="d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.234:8000/ready\": dial tcp 10.134.0.234:8000: connect: connection refused" Oct 29 23:16:01.947765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.942061 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f"] Oct 29 23:16:01.947765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:01.942317 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" podUID="261e1e36-d96d-4e60-93cd-bab1e3e610d1" containerName="maas-api" containerID="cri-o://9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2" gracePeriod=30 Oct 29 23:16:02.074631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.074559 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:16:02.221355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.221292 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-home\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-home\") pod \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " Oct 29 23:16:02.221555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.221382 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tls-certs\") pod \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " Oct 29 23:16:02.221555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.221500 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-dshm\") pod \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " Oct 29 23:16:02.221917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.221889 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-kube-api-access-g82vr\" (UniqueName: \"kubernetes.io/projected/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-kube-api-access-g82vr\") pod \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " Oct 29 23:16:02.222025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.221982 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tmp-dir\") pod \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " Oct 29 23:16:02.222167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.222149 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-model-cache\") pod \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\" (UID: \"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3\") " Oct 29 23:16:02.223417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.223386 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-model-cache" pod "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" (UID: "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:02.223654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.223632 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-home" pod "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" (UID: "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:02.229785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.229750 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tmp-dir" pod "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" (UID: "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:02.230681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.230643 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tls-certs" pod "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" (UID: "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:02.233089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.233054 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-kube-api-access-g82vr" pod "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" (UID: "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3"). InnerVolumeSpecName "kube-api-access-g82vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:02.237165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.237131 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-dshm" pod "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" (UID: "d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:02.278168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.278142 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:16:02.334938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.330296 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.334938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.330333 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.334938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.330346 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.334938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.330360 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82vr\" (UniqueName: \"kubernetes.io/projected/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-kube-api-access-g82vr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.334938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.330376 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.334938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.330390 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.443613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.432235 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-metrics-certs\") pod \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " Oct 29 23:16:02.443613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.432375 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261e1e36-d96d-4e60-93cd-bab1e3e610d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261e1e36-d96d-4e60-93cd-bab1e3e610d1-openshift-service-ca\") pod \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " Oct 29 23:16:02.443613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.432526 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261e1e36-d96d-4e60-93cd-bab1e3e610d1-kube-api-access-cbpqq\" (UniqueName: \"kubernetes.io/projected/261e1e36-d96d-4e60-93cd-bab1e3e610d1-kube-api-access-cbpqq\") pod \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " Oct 29 23:16:02.443613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.432753 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls\") pod \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\" (UID: \"261e1e36-d96d-4e60-93cd-bab1e3e610d1\") " Oct 29 23:16:02.443613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.437937 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261e1e36-d96d-4e60-93cd-bab1e3e610d1-openshift-service-ca" pod "261e1e36-d96d-4e60-93cd-bab1e3e610d1" (UID: "261e1e36-d96d-4e60-93cd-bab1e3e610d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:16:02.466599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.452010 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls" pod "261e1e36-d96d-4e60-93cd-bab1e3e610d1" (UID: "261e1e36-d96d-4e60-93cd-bab1e3e610d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:02.466599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.457727 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-metrics-certs" pod "261e1e36-d96d-4e60-93cd-bab1e3e610d1" (UID: "261e1e36-d96d-4e60-93cd-bab1e3e610d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:02.466599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.460289 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261e1e36-d96d-4e60-93cd-bab1e3e610d1-kube-api-access-cbpqq" pod "261e1e36-d96d-4e60-93cd-bab1e3e610d1" (UID: "261e1e36-d96d-4e60-93cd-bab1e3e610d1"). InnerVolumeSpecName "kube-api-access-cbpqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:02.547603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.547077 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.547603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.547120 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261e1e36-d96d-4e60-93cd-bab1e3e610d1-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.547603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.547135 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbpqq\" (UniqueName: \"kubernetes.io/projected/261e1e36-d96d-4e60-93cd-bab1e3e610d1-kube-api-access-cbpqq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.547603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.547148 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261e1e36-d96d-4e60-93cd-bab1e3e610d1-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:02.562601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.554789 5736 generic.go:356] "Generic (PLEG): container finished" podID="d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" containerID="3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7" exitCode=0 Oct 29 23:16:02.562601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.555092 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" event={"ID":"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3","Type":"ContainerDied","Data":"3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7"} Oct 29 23:16:02.562601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.555153 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" event={"ID":"d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3","Type":"ContainerDied","Data":"69132e3f676fec56e23f5d47452b67c140c67c888d752f0dfc75f28502e75c76"} Oct 29 23:16:02.562601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.555178 5736 scope.go:122] "RemoveContainer" containerID="3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7" Oct 29 23:16:02.562601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.555402 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4" Oct 29 23:16:02.773028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.748917 5736 generic.go:356] "Generic (PLEG): container finished" podID="261e1e36-d96d-4e60-93cd-bab1e3e610d1" containerID="9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2" exitCode=0 Oct 29 23:16:02.773028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.750180 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" Oct 29 23:16:02.773028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.753794 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" event={"ID":"261e1e36-d96d-4e60-93cd-bab1e3e610d1","Type":"ContainerDied","Data":"9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2"} Oct 29 23:16:02.773028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.753855 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f" event={"ID":"261e1e36-d96d-4e60-93cd-bab1e3e610d1","Type":"ContainerDied","Data":"8caa9152e8dc5893db4f7f94dd68ea58db5156e18f72a3dba462530f0087115c"} Oct 29 23:16:02.803034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.800437 5736 scope.go:122] "RemoveContainer" containerID="3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7" Oct 29 23:16:02.803034 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:16:02.800893 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7\": container with ID starting with 3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7 not found: ID does not exist" containerID="3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7" Oct 29 23:16:02.803034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.800938 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7"} err="failed to get container status \"3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7\": rpc error: code = NotFound desc = could not find container \"3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7\": container with ID starting with 3baa14a7a4a26f807c86ac1879b73be1daf57e702f92631184c9a4a06558cbc7 not found: ID does not exist" Oct 29 23:16:02.803034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.801255 5736 scope.go:122] "RemoveContainer" containerID="9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2" Oct 29 23:16:02.823625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.823227 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4"] Oct 29 23:16:02.829337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.829318 5736 scope.go:122] "RemoveContainer" containerID="9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2" Oct 29 23:16:02.829463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.829386 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c47a72/explicit-model-c47a72-kserve-5cff444c55-ck4w4"] Oct 29 23:16:02.829721 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:16:02.829699 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2\": container with ID starting with 9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2 not found: ID does not exist" containerID="9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2" Oct 29 23:16:02.829823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.829737 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2"} err="failed to get container status \"9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2\": rpc error: code = NotFound desc = could not find container \"9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2\": container with ID starting with 9c5d0d1da99fa0b75ff05e372bd35c782aae932c6eeded8c8f3e53de419298b2 not found: ID does not exist" Oct 29 23:16:02.865598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.864919 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f"] Oct 29 23:16:02.871875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:02.871777 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c47a72-65d87b7cc-lb47f"] Oct 29 23:16:03.009445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:03.009407 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261e1e36-d96d-4e60-93cd-bab1e3e610d1" path="/var/lib/kubelet/pods/261e1e36-d96d-4e60-93cd-bab1e3e610d1/volumes" Oct 29 23:16:03.010229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:03.010209 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" path="/var/lib/kubelet/pods/d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3/volumes" Oct 29 23:16:03.030419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:03.030382 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d"] Oct 29 23:16:03.034818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:03.034788 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d30db0-9cp8d"] Oct 29 23:16:03.159921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:03.159889 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc"] Oct 29 23:16:03.168863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:03.168831 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c47a72-rwzvc"] Oct 29 23:16:05.006551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.006517 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6851d12c-f899-4e6f-b024-99f2d1d9b449" path="/var/lib/kubelet/pods/6851d12c-f899-4e6f-b024-99f2d1d9b449/volumes" Oct 29 23:16:05.007054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.007037 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913373b1-8a1b-466a-8f5b-1a365257f8b2" path="/var/lib/kubelet/pods/913373b1-8a1b-466a-8f5b-1a365257f8b2/volumes" Oct 29 23:16:05.086139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.086098 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g"] Oct 29 23:16:05.086828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.086811 5736 cpu_manager.go:414] "containerMap: removing container" podUID="261e1e36-d96d-4e60-93cd-bab1e3e610d1" containerName="maas-api" Oct 29 23:16:05.086828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.086828 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261e1e36-d96d-4e60-93cd-bab1e3e610d1" containerName="maas-api" Oct 29 23:16:05.086934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.086861 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" containerName="main" Oct 29 23:16:05.086934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.086870 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" containerName="main" Oct 29 23:16:05.087007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.086989 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5508227-0dc5-4a0d-819f-fb2a6f8a2ec3" containerName="main" Oct 29 23:16:05.087007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.087006 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="261e1e36-d96d-4e60-93cd-bab1e3e610d1" containerName="maas-api" Oct 29 23:16:05.090788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.090766 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.113847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.113800 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g"] Oct 29 23:16:05.117966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.117926 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c83b50\"/\"openshift-service-ca.crt\"" Oct 29 23:16:05.117966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.117958 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c83b50\"/\"kube-root-ca.crt\"" Oct 29 23:16:05.118221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.118058 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c83b50\"/\"auth-test-model-c83b50-kserve-self-signed-certs\"" Oct 29 23:16:05.118221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.118100 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c83b50\"/\"default-dockercfg-2v7tb\"" Oct 29 23:16:05.197498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.197455 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872zs\" (UniqueName: \"kubernetes.io/projected/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-kube-api-access-872zs\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.197752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.197527 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tmp-dir\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.197752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.197662 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-model-cache\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.197901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.197826 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-dshm\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.197949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.197901 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tls-certs\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.198077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.198054 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-home\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.299689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.299598 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-model-cache\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.299869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.299730 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-dshm\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.299869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.299797 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tls-certs\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.299986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.299903 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-home\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.300045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.300028 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-model-cache\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.300142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.300122 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872zs\" (UniqueName: \"kubernetes.io/projected/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-kube-api-access-872zs\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.300201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.300175 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tmp-dir\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.300549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.300518 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-home\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.300719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.300603 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tmp-dir\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.302510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.302487 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-dshm\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.303096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.303071 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tls-certs\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.322921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.322894 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872zs\" (UniqueName: \"kubernetes.io/projected/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-kube-api-access-872zs\") pod \"auth-test-model-c83b50-kserve-66f4f969f7-ftg4g\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.406649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.406609 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:05.924494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:05.924468 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g"] Oct 29 23:16:05.927780 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:16:05.927745 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92ff4b26_e8fe_4f49_9b5b_01df9b2e16c7.slice/crio-18c358e55ba3306cc7869e271ec52c41f8bda9b1290ae7690654569eff009865 WatchSource:0}: Error finding container 18c358e55ba3306cc7869e271ec52c41f8bda9b1290ae7690654569eff009865: Status 404 returned error can't find the container with id 18c358e55ba3306cc7869e271ec52c41f8bda9b1290ae7690654569eff009865 Oct 29 23:16:06.796927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:06.796886 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" event={"ID":"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7","Type":"ContainerStarted","Data":"d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0"} Oct 29 23:16:06.796927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:06.796935 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" event={"ID":"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7","Type":"ContainerStarted","Data":"18c358e55ba3306cc7869e271ec52c41f8bda9b1290ae7690654569eff009865"} Oct 29 23:16:06.841316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:06.841257 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" podStartSLOduration=1.841237731 podStartE2EDuration="1.841237731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:06.837422379 +0000 UTC m=+1963.953341987" watchObservedRunningTime="2026-10-29 23:16:06.841237731 +0000 UTC m=+1963.957157359" Oct 29 23:16:07.414096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:07.414064 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:08.282094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.218651 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w"] Oct 29 23:16:08.333599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.332791 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w"] Oct 29 23:16:08.333599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.332925 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:08.369602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.368842 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-2b0108\"" Oct 29 23:16:08.432668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.427306 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-2b0108-dockercfg-ct59l\"" Oct 29 23:16:08.446949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.446897 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mwwn\" (UniqueName: \"kubernetes.io/projected/9fbc02cd-621e-4251-a0c7-4835484302c7-kube-api-access-7mwwn\") pod \"payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w\" (UID: \"9fbc02cd-621e-4251-a0c7-4835484302c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:08.447409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.447385 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc02cd-621e-4251-a0c7-4835484302c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w\" (UID: \"9fbc02cd-621e-4251-a0c7-4835484302c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:08.549107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.549025 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mwwn\" (UniqueName: \"kubernetes.io/projected/9fbc02cd-621e-4251-a0c7-4835484302c7-kube-api-access-7mwwn\") pod \"payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w\" (UID: \"9fbc02cd-621e-4251-a0c7-4835484302c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:08.549284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.549237 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc02cd-621e-4251-a0c7-4835484302c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w\" (UID: \"9fbc02cd-621e-4251-a0c7-4835484302c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:08.549930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.549907 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc02cd-621e-4251-a0c7-4835484302c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w\" (UID: \"9fbc02cd-621e-4251-a0c7-4835484302c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:08.574405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.574372 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mwwn\" (UniqueName: \"kubernetes.io/projected/9fbc02cd-621e-4251-a0c7-4835484302c7-kube-api-access-7mwwn\") pod \"payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w\" (UID: \"9fbc02cd-621e-4251-a0c7-4835484302c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:08.756896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:08.756867 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:09.071285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:09.071259 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w"] Oct 29 23:16:09.074274 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:16:09.074247 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbc02cd_621e_4251_a0c7_4835484302c7.slice/crio-79073c43c39dd251cf66188595546a85f4897f610db5b9e0e08dd9b4290dd02d WatchSource:0}: Error finding container 79073c43c39dd251cf66188595546a85f4897f610db5b9e0e08dd9b4290dd02d: Status 404 returned error can't find the container with id 79073c43c39dd251cf66188595546a85f4897f610db5b9e0e08dd9b4290dd02d Oct 29 23:16:09.838653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:09.838612 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" event={"ID":"9fbc02cd-621e-4251-a0c7-4835484302c7","Type":"ContainerStarted","Data":"59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8"} Oct 29 23:16:09.839124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:09.838673 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" event={"ID":"9fbc02cd-621e-4251-a0c7-4835484302c7","Type":"ContainerStarted","Data":"79073c43c39dd251cf66188595546a85f4897f610db5b9e0e08dd9b4290dd02d"} Oct 29 23:16:09.839124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:09.838714 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:09.882428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:09.882372 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" podStartSLOduration=1.882359075 podStartE2EDuration="1.882359075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:09.878776943 +0000 UTC m=+1966.994696550" watchObservedRunningTime="2026-10-29 23:16:09.882359075 +0000 UTC m=+1966.998278688" Oct 29 23:16:10.522646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:10.520720 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr"] Oct 29 23:16:10.522646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:10.520980 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" containerID="cri-o://663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1" gracePeriod=30 Oct 29 23:16:13.031862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:13.031829 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr"] Oct 29 23:16:13.043882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:13.043693 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f215939-f9wvr"] Oct 29 23:16:14.031323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:14.031286 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg"] Oct 29 23:16:14.041712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:14.041683 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de852370-llflg"] Oct 29 23:16:15.029351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.029308 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0304ee61-d364-4f68-afe6-04af71b0e1df" path="/var/lib/kubelet/pods/0304ee61-d364-4f68-afe6-04af71b0e1df/volumes" Oct 29 23:16:15.030211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.030177 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d460e938-bd5b-42e4-ad89-f6d8a75e35df" path="/var/lib/kubelet/pods/d460e938-bd5b-42e4-ad89-f6d8a75e35df/volumes" Oct 29 23:16:15.413884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.413794 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:15.431568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.431539 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:15.840057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.840027 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:16:15.897244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.897210 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2454278-070f-4050-9906-5e030d277bc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2454278-070f-4050-9906-5e030d277bc3-istio-podinfo\") pod \"e2454278-070f-4050-9906-5e030d277bc3\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " Oct 29 23:16:15.897653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.897619 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-socket\") pod \"e2454278-070f-4050-9906-5e030d277bc3\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " Oct 29 23:16:15.897776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.897677 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2454278-070f-4050-9906-5e030d277bc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2454278-070f-4050-9906-5e030d277bc3-istiod-ca-cert\") pod \"e2454278-070f-4050-9906-5e030d277bc3\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " Oct 29 23:16:15.897776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.897740 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-credential-socket\") pod \"e2454278-070f-4050-9906-5e030d277bc3\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " Oct 29 23:16:15.897776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.897769 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-envoy\") pod \"e2454278-070f-4050-9906-5e030d277bc3\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " Oct 29 23:16:15.897929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.897914 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-istio-token\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-istio-token\") pod \"e2454278-070f-4050-9906-5e030d277bc3\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " Oct 29 23:16:15.897980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.897965 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-certs\") pod \"e2454278-070f-4050-9906-5e030d277bc3\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " Oct 29 23:16:15.898099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.898085 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-data\") pod \"e2454278-070f-4050-9906-5e030d277bc3\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " Oct 29 23:16:15.898190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.898176 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-kube-api-access-drhjb\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-kube-api-access-drhjb\") pod \"e2454278-070f-4050-9906-5e030d277bc3\" (UID: \"e2454278-070f-4050-9906-5e030d277bc3\") " Oct 29 23:16:15.901001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.899872 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2454278-070f-4050-9906-5e030d277bc3-istio-podinfo" pod "e2454278-070f-4050-9906-5e030d277bc3" (UID: "e2454278-070f-4050-9906-5e030d277bc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:16:15.901001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.900502 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-certs" pod "e2454278-070f-4050-9906-5e030d277bc3" (UID: "e2454278-070f-4050-9906-5e030d277bc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:15.901001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.900554 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-data" pod "e2454278-070f-4050-9906-5e030d277bc3" (UID: "e2454278-070f-4050-9906-5e030d277bc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:15.901001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.900720 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2454278-070f-4050-9906-5e030d277bc3-istiod-ca-cert" pod "e2454278-070f-4050-9906-5e030d277bc3" (UID: "e2454278-070f-4050-9906-5e030d277bc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:16:15.901001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.900806 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-credential-socket" pod "e2454278-070f-4050-9906-5e030d277bc3" (UID: "e2454278-070f-4050-9906-5e030d277bc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:15.901001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.900976 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-socket" pod "e2454278-070f-4050-9906-5e030d277bc3" (UID: "e2454278-070f-4050-9906-5e030d277bc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:15.902019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.901461 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-kube-api-access-drhjb" pod "e2454278-070f-4050-9906-5e030d277bc3" (UID: "e2454278-070f-4050-9906-5e030d277bc3"). InnerVolumeSpecName "kube-api-access-drhjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:15.910626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.903115 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-istio-token" pod "e2454278-070f-4050-9906-5e030d277bc3" (UID: "e2454278-070f-4050-9906-5e030d277bc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:15.910626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.903155 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-envoy" pod "e2454278-070f-4050-9906-5e030d277bc3" (UID: "e2454278-070f-4050-9906-5e030d277bc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:15.919188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.917938 5736 generic.go:356] "Generic (PLEG): container finished" podID="e2454278-070f-4050-9906-5e030d277bc3" containerID="663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1" exitCode=0 Oct 29 23:16:15.919188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.918034 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" Oct 29 23:16:15.919188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.918089 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" event={"ID":"e2454278-070f-4050-9906-5e030d277bc3","Type":"ContainerDied","Data":"663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1"} Oct 29 23:16:15.919188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.918169 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr" event={"ID":"e2454278-070f-4050-9906-5e030d277bc3","Type":"ContainerDied","Data":"305d65a1c469c00064b5ff51949a54fdeeac157b9cbda7eb2e38ad1d5d49fa4b"} Oct 29 23:16:15.919188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.918207 5736 scope.go:122] "RemoveContainer" containerID="663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1" Oct 29 23:16:15.941382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.941356 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:16:15.954517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.954491 5736 scope.go:122] "RemoveContainer" containerID="c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453" Oct 29 23:16:15.968759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.968737 5736 scope.go:122] "RemoveContainer" containerID="663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1" Oct 29 23:16:15.969065 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:16:15.969044 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1\": container with ID starting with 663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1 not found: ID does not exist" containerID="663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1" Oct 29 23:16:15.969152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.969073 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1"} err="failed to get container status \"663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1\": rpc error: code = NotFound desc = could not find container \"663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1\": container with ID starting with 663f38739435c8174da8a1cbef4746d87ebea1b9fac529b8a899334fd4f5b6d1 not found: ID does not exist" Oct 29 23:16:15.969152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.969092 5736 scope.go:122] "RemoveContainer" containerID="c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453" Oct 29 23:16:15.969367 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:16:15.969349 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453\": container with ID starting with c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453 not found: ID does not exist" containerID="c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453" Oct 29 23:16:15.969419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:15.969372 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453"} err="failed to get container status \"c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453\": rpc error: code = NotFound desc = could not find container \"c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453\": container with ID starting with c7bdf639625eb94a88453d7cef8dfffacdce07e0d02076f0ff1493318cbbe453 not found: ID does not exist" Oct 29 23:16:16.001088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.001055 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2454278-070f-4050-9906-5e030d277bc3-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:16.001088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.001088 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:16.001253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.001104 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2454278-070f-4050-9906-5e030d277bc3-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:16.001253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.001120 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:16.001253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.001135 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:16.001253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.001148 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:16.001253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.001162 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:16.001253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.001183 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2454278-070f-4050-9906-5e030d277bc3-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:16.001253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.001196 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drhjb\" (UniqueName: \"kubernetes.io/projected/e2454278-070f-4050-9906-5e030d277bc3-kube-api-access-drhjb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:16:16.025840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.025808 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr"] Oct 29 23:16:16.032574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:16.032546 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-c47a72-openshift-default-66cf4bcf57-6m5wr"] Oct 29 23:16:17.070599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:17.063065 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2454278-070f-4050-9906-5e030d277bc3" path="/var/lib/kubelet/pods/e2454278-070f-4050-9906-5e030d277bc3/volumes" Oct 29 23:16:20.871605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:20.871554 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:16:22.242146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.242109 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl"] Oct 29 23:16:22.242930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.242902 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" Oct 29 23:16:22.242930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.242927 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" Oct 29 23:16:22.243226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.243203 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" Oct 29 23:16:22.247743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.247722 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:22.266247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.266225 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl"] Oct 29 23:16:22.273878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.273856 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-a4f2db\"" Oct 29 23:16:22.508161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.508070 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67sxt\" (UniqueName: \"kubernetes.io/projected/c17d87bb-02f0-4574-9f71-b6c97321b347-kube-api-access-67sxt\") pod \"payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl\" (UID: \"c17d87bb-02f0-4574-9f71-b6c97321b347\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:22.508350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.508321 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17d87bb-02f0-4574-9f71-b6c97321b347-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl\" (UID: \"c17d87bb-02f0-4574-9f71-b6c97321b347\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:22.631729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.629643 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67sxt\" (UniqueName: \"kubernetes.io/projected/c17d87bb-02f0-4574-9f71-b6c97321b347-kube-api-access-67sxt\") pod \"payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl\" (UID: \"c17d87bb-02f0-4574-9f71-b6c97321b347\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:22.631729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.629849 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17d87bb-02f0-4574-9f71-b6c97321b347-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl\" (UID: \"c17d87bb-02f0-4574-9f71-b6c97321b347\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:22.631729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.631038 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17d87bb-02f0-4574-9f71-b6c97321b347-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl\" (UID: \"c17d87bb-02f0-4574-9f71-b6c97321b347\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:22.705603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.699402 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67sxt\" (UniqueName: \"kubernetes.io/projected/c17d87bb-02f0-4574-9f71-b6c97321b347-kube-api-access-67sxt\") pod \"payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl\" (UID: \"c17d87bb-02f0-4574-9f71-b6c97321b347\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:22.863362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:22.863273 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:23.183178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:23.183154 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl"] Oct 29 23:16:23.184989 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:16:23.184962 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17d87bb_02f0_4574_9f71_b6c97321b347.slice/crio-34f94e9cd4595f25ba429b996ea1163ff05c7a8766dfd4459df5ed8bd70866f9 WatchSource:0}: Error finding container 34f94e9cd4595f25ba429b996ea1163ff05c7a8766dfd4459df5ed8bd70866f9: Status 404 returned error can't find the container with id 34f94e9cd4595f25ba429b996ea1163ff05c7a8766dfd4459df5ed8bd70866f9 Oct 29 23:16:24.027086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:24.027051 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" event={"ID":"c17d87bb-02f0-4574-9f71-b6c97321b347","Type":"ContainerStarted","Data":"afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192"} Oct 29 23:16:24.027525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:24.027135 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" event={"ID":"c17d87bb-02f0-4574-9f71-b6c97321b347","Type":"ContainerStarted","Data":"34f94e9cd4595f25ba429b996ea1163ff05c7a8766dfd4459df5ed8bd70866f9"} Oct 29 23:16:24.027525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:24.027286 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:24.073880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:24.073822 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" podStartSLOduration=2.073804373 podStartE2EDuration="2.073804373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:24.069542484 +0000 UTC m=+1981.185462090" watchObservedRunningTime="2026-10-29 23:16:24.073804373 +0000 UTC m=+1981.189723980" Oct 29 23:16:25.534034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:25.534001 5736 scope.go:122] "RemoveContainer" containerID="df6db142fbba4f24054b1fe4820f4b2dd622635d6078ffd9fdca74c9a5c4a40f" Oct 29 23:16:25.545378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:25.545355 5736 scope.go:122] "RemoveContainer" containerID="775ca79b60f63559089a36141b242fe7057095843b938d66597c4c7212245f58" Oct 29 23:16:25.558119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:25.558096 5736 scope.go:122] "RemoveContainer" containerID="92012f169ee8d0a687c05760d7d4cfba41486c6e41249e014ee7d7c35363b470" Oct 29 23:16:26.831114 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:16:26.831070 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:16:28.058911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:28.058652 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz"] Oct 29 23:16:28.089703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:28.089631 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj"] Oct 29 23:16:28.098239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:28.098195 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c86224-x92fz"] Oct 29 23:16:28.101638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:28.101601 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc91e03f-qxqxj"] Oct 29 23:16:29.008018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:29.007987 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae34499a-24fe-45bc-91db-f2f3efdd9af7" path="/var/lib/kubelet/pods/ae34499a-24fe-45bc-91db-f2f3efdd9af7/volumes" Oct 29 23:16:29.008525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:29.008510 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db05b052-2c44-4e6b-950e-a4f1c425e87c" path="/var/lib/kubelet/pods/db05b052-2c44-4e6b-950e-a4f1c425e87c/volumes" Oct 29 23:16:33.306253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.306209 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5"] Oct 29 23:16:33.307241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.307214 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" Oct 29 23:16:33.307241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.307240 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" Oct 29 23:16:33.307468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.307453 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2454278-070f-4050-9906-5e030d277bc3" containerName="istio-proxy" Oct 29 23:16:33.314772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.314751 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.336755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.335079 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5"] Oct 29 23:16:33.342442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.342382 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-52f67b\"/\"openshift-service-ca.crt\"" Oct 29 23:16:33.342442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.342398 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-52f67b\"/\"kube-root-ca.crt\"" Oct 29 23:16:33.342846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.342808 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-52f67b\"/\"auth-test-model-52f67b-kserve-self-signed-certs\"" Oct 29 23:16:33.342944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.342927 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-52f67b\"/\"default-dockercfg-m7qwz\"" Oct 29 23:16:33.413264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.412625 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tls-certs\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.413264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.412675 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-home\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.413264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.412719 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-dshm\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.413264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.412970 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tmp-dir\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.413264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.413005 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjn8b\" (UniqueName: \"kubernetes.io/projected/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-kube-api-access-mjn8b\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.413264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.413061 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-model-cache\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.514144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.514113 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-dshm\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.514432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.514414 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tmp-dir\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.514521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.514457 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjn8b\" (UniqueName: \"kubernetes.io/projected/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-kube-api-access-mjn8b\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.514609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.514522 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-model-cache\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.514682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.514607 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tls-certs\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.514682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.514644 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-home\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.514935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.514871 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tmp-dir\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.515042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.515012 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-home\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.515663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.515640 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-model-cache\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.516638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.516565 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-dshm\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.519187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.519161 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tls-certs\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.584045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.583965 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjn8b\" (UniqueName: \"kubernetes.io/projected/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-kube-api-access-mjn8b\") pod \"auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:33.644934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:33.644897 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:34.071546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.071517 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5"] Oct 29 23:16:34.074338 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:16:34.074312 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6a3bd94_e937_433e_9d7c_6ae6fe8f4be9.slice/crio-9b1d6cffa8a587d3814b28f1ee7edd2953e441ae7387753df65d358afaf69e10 WatchSource:0}: Error finding container 9b1d6cffa8a587d3814b28f1ee7edd2953e441ae7387753df65d358afaf69e10: Status 404 returned error can't find the container with id 9b1d6cffa8a587d3814b28f1ee7edd2953e441ae7387753df65d358afaf69e10 Oct 29 23:16:34.153350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.153312 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" event={"ID":"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9","Type":"ContainerStarted","Data":"1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9"} Oct 29 23:16:34.153499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.153367 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" event={"ID":"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9","Type":"ContainerStarted","Data":"9b1d6cffa8a587d3814b28f1ee7edd2953e441ae7387753df65d358afaf69e10"} Oct 29 23:16:34.199533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.199469 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" podStartSLOduration=1.199451408 podStartE2EDuration="1.199451408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:34.195951325 +0000 UTC m=+1991.311870931" watchObservedRunningTime="2026-10-29 23:16:34.199451408 +0000 UTC m=+1991.315371015" Oct 29 23:16:34.645555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.645521 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:34.735430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.735400 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj"] Oct 29 23:16:34.742379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.742357 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.767003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.766955 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2126fb-openshift-default-dockercfg-2swzx\"" Oct 29 23:16:34.811206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.811167 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj"] Oct 29 23:16:34.885953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.885915 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-data\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.886152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.886031 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-token\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.886152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.886099 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-podinfo\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.886296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.886157 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-envoy\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.886296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.886182 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-credential-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.886296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.886197 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddsb\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-kube-api-access-dddsb\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.886466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.886353 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-certs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.886466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.886449 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.886610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.886475 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09ce4d3-71b2-458f-ad73-542a86c10268-istiod-ca-cert\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988326 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-data\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988395 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-token\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988422 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-podinfo\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988476 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-envoy\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988509 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-credential-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988535 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddsb\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-kube-api-access-dddsb\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988629 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-certs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988712 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988751 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09ce4d3-71b2-458f-ad73-542a86c10268-istiod-ca-cert\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.988784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988770 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-data\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.989056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.988951 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-credential-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.989056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.989012 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-certs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.989182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.989060 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.989482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.989462 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09ce4d3-71b2-458f-ad73-542a86c10268-istiod-ca-cert\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.990812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.990790 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-envoy\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:34.991098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:34.991082 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-podinfo\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:35.011727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:35.011689 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-token\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:35.011857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:35.011832 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddsb\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-kube-api-access-dddsb\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:35.040452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:35.040428 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:16:35.064471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:35.064447 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:35.400196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:35.399895 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj"] Oct 29 23:16:35.410429 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:16:35.410399 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09ce4d3_71b2_458f_ad73_542a86c10268.slice/crio-15971c90222b6008d63bdbee513903b597d093f0f005f602a0a97f7e011b702b WatchSource:0}: Error finding container 15971c90222b6008d63bdbee513903b597d093f0f005f602a0a97f7e011b702b: Status 404 returned error can't find the container with id 15971c90222b6008d63bdbee513903b597d093f0f005f602a0a97f7e011b702b Oct 29 23:16:35.416104 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:35.416072 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:16:35.416204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:35.416138 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:16:35.416204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:35.416171 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:16:36.177193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:36.177146 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" event={"ID":"f09ce4d3-71b2-458f-ad73-542a86c10268","Type":"ContainerStarted","Data":"e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289"} Oct 29 23:16:36.177615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:36.177227 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" event={"ID":"f09ce4d3-71b2-458f-ad73-542a86c10268","Type":"ContainerStarted","Data":"15971c90222b6008d63bdbee513903b597d093f0f005f602a0a97f7e011b702b"} Oct 29 23:16:36.260020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:36.259949 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" podStartSLOduration=2.259925269 podStartE2EDuration="2.259925269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:36.242095762 +0000 UTC m=+1993.358015368" watchObservedRunningTime="2026-10-29 23:16:36.259925269 +0000 UTC m=+1993.375844876" Oct 29 23:16:37.064918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:37.064881 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:37.066765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:37.066737 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:16:37.066900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:37.066793 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:16:38.065591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:38.065546 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:16:38.066007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:38.065633 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:16:39.065172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:39.065131 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:16:39.065365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:39.065194 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:16:40.064960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:40.064918 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:16:40.065456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:40.064980 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:16:41.065098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:41.065066 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:16:41.065561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:41.065533 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:16:42.032245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:42.032211 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h"] Oct 29 23:16:42.044130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:42.044092 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b310a3-tvg2h"] Oct 29 23:16:42.070550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:42.070522 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:42.071178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:42.071152 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:42.072147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:42.072120 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:16:43.006281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:43.006237 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f" path="/var/lib/kubelet/pods/badf11f4-bac4-4a7e-8d7d-0b3f49c31d0f/volumes" Oct 29 23:16:43.645683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:43.645645 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:43.659038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:43.659008 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:44.301972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:44.301940 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:16:46.031001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:46.030966 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj"] Oct 29 23:16:46.041787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:46.041754 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b933b9-r5rpj"] Oct 29 23:16:47.006337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:47.006303 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fdcf95e-b463-494d-8ebc-5b04b812fb19" path="/var/lib/kubelet/pods/2fdcf95e-b463-494d-8ebc-5b04b812fb19/volumes" Oct 29 23:16:47.030761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:47.030736 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5"] Oct 29 23:16:47.042050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:47.042023 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eff51a-h5bw5"] Oct 29 23:16:48.256492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.256452 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm"] Oct 29 23:16:48.261459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.261439 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.285625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.285564 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm"] Oct 29 23:16:48.290105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.290054 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-model-cache\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.290405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.290382 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-dshm\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.290538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.290517 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tls-certs\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.290635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.290551 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzpz\" (UniqueName: \"kubernetes.io/projected/e97569b6-4ebf-4501-9897-a8c34b5c68ef-kube-api-access-vvzpz\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.290635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.290619 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tmp-dir\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.290731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.290697 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-home\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.291686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.291666 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-8a8b61\"/\"openshift-service-ca.crt\"" Oct 29 23:16:48.291812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.291758 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-8a8b61\"/\"kube-root-ca.crt\"" Oct 29 23:16:48.291892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.291846 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-8a8b61\"/\"auth-test-model-8a8b61-kserve-self-signed-certs\"" Oct 29 23:16:48.292768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.292732 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-8a8b61\"/\"default-dockercfg-qz57w\"" Oct 29 23:16:48.391812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.391785 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tls-certs\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.391812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.391817 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzpz\" (UniqueName: \"kubernetes.io/projected/e97569b6-4ebf-4501-9897-a8c34b5c68ef-kube-api-access-vvzpz\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.392083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.391865 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tmp-dir\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.392083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.391940 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-home\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.392083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.392037 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-model-cache\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.392260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.392093 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-dshm\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.392620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.392548 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-model-cache\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.392620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.392599 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-home\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.392826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.392761 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tmp-dir\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.397595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.395121 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-dshm\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.397595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.395128 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tls-certs\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.418183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.418157 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzpz\" (UniqueName: \"kubernetes.io/projected/e97569b6-4ebf-4501-9897-a8c34b5c68ef-kube-api-access-vvzpz\") pod \"auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:48.579220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:48.578894 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:49.008720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:49.008678 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53b6ac4-e35b-43fa-91cc-f95db9196fc9" path="/var/lib/kubelet/pods/e53b6ac4-e35b-43fa-91cc-f95db9196fc9/volumes" Oct 29 23:16:49.042851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:49.042821 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm"] Oct 29 23:16:49.044204 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:16:49.044181 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode97569b6_4ebf_4501_9897_a8c34b5c68ef.slice/crio-aaef4bc65cce813f5708f1622dfd3157c359828f0a8452f859c3bbd40541f4f9 WatchSource:0}: Error finding container aaef4bc65cce813f5708f1622dfd3157c359828f0a8452f859c3bbd40541f4f9: Status 404 returned error can't find the container with id aaef4bc65cce813f5708f1622dfd3157c359828f0a8452f859c3bbd40541f4f9 Oct 29 23:16:49.342195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:49.342096 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" event={"ID":"e97569b6-4ebf-4501-9897-a8c34b5c68ef","Type":"ContainerStarted","Data":"e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be"} Oct 29 23:16:49.342195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:49.342146 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" event={"ID":"e97569b6-4ebf-4501-9897-a8c34b5c68ef","Type":"ContainerStarted","Data":"aaef4bc65cce813f5708f1622dfd3157c359828f0a8452f859c3bbd40541f4f9"} Oct 29 23:16:49.387317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:49.387262 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" podStartSLOduration=1.387249637 podStartE2EDuration="1.387249637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:49.383102413 +0000 UTC m=+2006.499022018" watchObservedRunningTime="2026-10-29 23:16:49.387249637 +0000 UTC m=+2006.503169239" Oct 29 23:16:49.580655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:49.580571 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:56.831728 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:16:56.831640 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:16:58.579711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:58.579677 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:58.592732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:58.592703 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:16:59.484539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:16:59.484509 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:17:00.056602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:00.055629 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp"] Oct 29 23:17:00.072602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:00.070008 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09f7f54-b6pxp"] Oct 29 23:17:01.005620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:01.005572 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="632ff5ad-1de4-4dce-b771-bef6c4b28731" path="/var/lib/kubelet/pods/632ff5ad-1de4-4dce-b771-bef6c4b28731/volumes" Oct 29 23:17:06.084423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:06.084382 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn"] Oct 29 23:17:06.093325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:06.093285 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381f1999-lwznn"] Oct 29 23:17:07.006680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:07.006647 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35acf06c-b040-4178-88dd-d48cc6d6291d" path="/var/lib/kubelet/pods/35acf06c-b040-4178-88dd-d48cc6d6291d/volumes" Oct 29 23:17:11.030459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:11.030414 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v"] Oct 29 23:17:11.034345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:11.034320 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a041ba13-44n8v"] Oct 29 23:17:12.029503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:12.029472 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw"] Oct 29 23:17:12.041237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:12.041210 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ef3bb3e-qw7lw"] Oct 29 23:17:13.006028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:13.005991 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f64ca3e-213a-473b-8722-27255ecbd7ea" path="/var/lib/kubelet/pods/0f64ca3e-213a-473b-8722-27255ecbd7ea/volumes" Oct 29 23:17:13.006564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:13.006537 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da536371-ef7f-438b-9124-185a7b439107" path="/var/lib/kubelet/pods/da536371-ef7f-438b-9124-185a7b439107/volumes" Oct 29 23:17:25.627197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:25.627156 5736 scope.go:122] "RemoveContainer" containerID="2bb0ef51aa902eb315aa2203b1b5cc8772a87d0af00d3afb5edc081c82ede1f7" Oct 29 23:17:25.642055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:25.642026 5736 scope.go:122] "RemoveContainer" containerID="c9aeefac7bb620dad9750387f66c9f9041239780397fae883582c233b670d95e" Oct 29 23:17:25.678688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:25.678662 5736 scope.go:122] "RemoveContainer" containerID="01e7081e9a68f220bc0de16242194ae135f73dc223130a698885d594ea5932e6" Oct 29 23:17:25.692882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:25.692847 5736 scope.go:122] "RemoveContainer" containerID="4b7c89e14364fdac084c2a411667f10458c370eeeaf5db1b2edf68a350b21084" Oct 29 23:17:25.706912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:25.706890 5736 scope.go:122] "RemoveContainer" containerID="53105424a63f9fe62b5b22e1cb4448b77ab342859329563094debf533ba448f1" Oct 29 23:17:25.717941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:25.717919 5736 scope.go:122] "RemoveContainer" containerID="74108cb99f3358a5cf9ec8ad20ca79a24259689b8df5af78b8ba8ffa64cea6ff" Oct 29 23:17:25.733257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:25.731921 5736 scope.go:122] "RemoveContainer" containerID="25f29bebf9fbf5b9e04c6a603c350ef93aefd523547c8377b81eb62a1449a104" Oct 29 23:17:25.751811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:25.751703 5736 scope.go:122] "RemoveContainer" containerID="0919ad9e51dab8d6243ec48b7a2cf1de533ac08ea2a801180aecefe5e5ed49e4" Oct 29 23:17:26.058909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:26.058871 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2"] Oct 29 23:17:26.092641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:26.092608 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79384a5b-csgr2"] Oct 29 23:17:26.125285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:26.125125 5736 scope.go:122] "RemoveContainer" containerID="69fe2b5850205513e0323cb25237fc3b33f8e00ccb5fa7035374863db141e335" Oct 29 23:17:26.142234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:26.142206 5736 scope.go:122] "RemoveContainer" containerID="18ce5ddfceb5f3dac47f396f1453bb50a03eea87f5f00a033dc293b5fbe445d7" Oct 29 23:17:26.156047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:26.156024 5736 scope.go:122] "RemoveContainer" containerID="a16dca94633f7eabe3b61478f9d47ad9a71f937115b6137b4ac41627fe100d05" Oct 29 23:17:26.166469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:26.166442 5736 scope.go:122] "RemoveContainer" containerID="5ea7e520697e2a9f9f0221a2f7bbdc2a651b052ed7aa369f14aa87ef0a326269" Oct 29 23:17:26.181954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:26.181928 5736 scope.go:122] "RemoveContainer" containerID="f5967c2218914f848041dedfcc35655f4ee3e18620cfcec1a673ed65fbd4263e" Oct 29 23:17:26.194777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:26.194753 5736 scope.go:122] "RemoveContainer" containerID="3284d06e79e8279f4726b307b4548cb3b7246b81d57002a559f6ca2eb85aa5fe" Oct 29 23:17:26.827730 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:17:26.827692 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:17:27.009392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:27.008869 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f59b8f7-354c-434c-acd6-401791fd2bec" path="/var/lib/kubelet/pods/3f59b8f7-354c-434c-acd6-401791fd2bec/volumes" Oct 29 23:17:49.886399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:49.886364 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s"] Oct 29 23:17:49.890737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:49.890716 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:49.918986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:49.918670 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:17:49.918986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:49.918685 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:17:49.924064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:49.924037 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s"] Oct 29 23:17:49.970538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:49.970505 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6sk\" (UniqueName: \"kubernetes.io/projected/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-kube-api-access-fd6sk\") pod \"maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:49.970746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:49.970603 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:49.970830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:49.970798 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:50.072241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:50.072206 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6sk\" (UniqueName: \"kubernetes.io/projected/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-kube-api-access-fd6sk\") pod \"maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:50.072415 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:50.072332 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:50.072415 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:50.072393 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:50.073092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:50.073065 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:50.074942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:50.074924 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:50.095536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:50.095503 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6sk\" (UniqueName: \"kubernetes.io/projected/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-kube-api-access-fd6sk\") pod \"maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:50.205620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:50.205511 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:50.515793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:50.515692 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s"] Oct 29 23:17:50.519523 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:17:50.519487 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed5370d_2a2d_44b0_9ef0_1d4d683fe447.slice/crio-4636bf7b65d9bbff6b408d9686a2b42f98147d85730277107826ab0cb11052eb WatchSource:0}: Error finding container 4636bf7b65d9bbff6b408d9686a2b42f98147d85730277107826ab0cb11052eb: Status 404 returned error can't find the container with id 4636bf7b65d9bbff6b408d9686a2b42f98147d85730277107826ab0cb11052eb Oct 29 23:17:51.450674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:51.450631 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" event={"ID":"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447","Type":"ContainerStarted","Data":"5e117547f7b88f244bef0cd73055a6c01477173406a229fe101ff35448695844"} Oct 29 23:17:51.451201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:51.450701 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" event={"ID":"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447","Type":"ContainerStarted","Data":"4636bf7b65d9bbff6b408d9686a2b42f98147d85730277107826ab0cb11052eb"} Oct 29 23:17:51.493915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:51.493820 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" podStartSLOduration=2.493805239 podStartE2EDuration="2.493805239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:51.490686308 +0000 UTC m=+2068.606605916" watchObservedRunningTime="2026-10-29 23:17:51.493805239 +0000 UTC m=+2068.609724844" Oct 29 23:17:52.460454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:52.460417 5736 generic.go:356] "Generic (PLEG): container finished" podID="7ed5370d-2a2d-44b0-9ef0-1d4d683fe447" containerID="5e117547f7b88f244bef0cd73055a6c01477173406a229fe101ff35448695844" exitCode=0 Oct 29 23:17:52.460928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:52.460501 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" event={"ID":"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447","Type":"ContainerDied","Data":"5e117547f7b88f244bef0cd73055a6c01477173406a229fe101ff35448695844"} Oct 29 23:17:53.643889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.643861 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:53.745063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.744984 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-kube-api-access-fd6sk\" (UniqueName: \"kubernetes.io/projected/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-kube-api-access-fd6sk\") pod \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " Oct 29 23:17:53.745267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.745245 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-xks-ca\") pod \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " Oct 29 23:17:53.745458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.745432 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-openshift-ca\") pod \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\" (UID: \"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447\") " Oct 29 23:17:53.745825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.745802 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-openshift-ca" pod "7ed5370d-2a2d-44b0-9ef0-1d4d683fe447" (UID: "7ed5370d-2a2d-44b0-9ef0-1d4d683fe447"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:17:53.746234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.746213 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:17:53.747343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.747318 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-kube-api-access-fd6sk" pod "7ed5370d-2a2d-44b0-9ef0-1d4d683fe447" (UID: "7ed5370d-2a2d-44b0-9ef0-1d4d683fe447"). InnerVolumeSpecName "kube-api-access-fd6sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:53.747426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.747378 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-xks-ca" pod "7ed5370d-2a2d-44b0-9ef0-1d4d683fe447" (UID: "7ed5370d-2a2d-44b0-9ef0-1d4d683fe447"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:17:53.847562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.847532 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6sk\" (UniqueName: \"kubernetes.io/projected/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-kube-api-access-fd6sk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:17:53.847562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:53.847562 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:17:54.478394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:54.478364 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" Oct 29 23:17:54.478567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:54.478397 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s" event={"ID":"7ed5370d-2a2d-44b0-9ef0-1d4d683fe447","Type":"ContainerDied","Data":"4636bf7b65d9bbff6b408d9686a2b42f98147d85730277107826ab0cb11052eb"} Oct 29 23:17:54.478567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:17:54.478482 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4636bf7b65d9bbff6b408d9686a2b42f98147d85730277107826ab0cb11052eb" Oct 29 23:17:56.826327 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:17:56.826290 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:18:08.455097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:08.455007 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc"] Oct 29 23:18:08.455549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:08.455248 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" containerID="cri-o://6084ba711f9bf011bd44a2344821a26ab04fccc57ce21888c03772fe26face17" gracePeriod=30 Oct 29 23:18:08.466459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:08.466425 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr"] Oct 29 23:18:08.466708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:08.466673 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" containerID="cri-o://9d5a8165f1f42601dee81bf079b5e98dc871de8d1585edfee9d5802fdfdb70ab" gracePeriod=30 Oct 29 23:18:08.470373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:08.470350 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5"] Oct 29 23:18:08.470520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:08.470502 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" containerID="cri-o://7b63381578a9dd6aad401db6b323e7b22e0dfdaaab4f20907edabc3080b0e1ba" gracePeriod=30 Oct 29 23:18:09.033259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:09.033226 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq"] Oct 29 23:18:09.037160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:09.037131 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742bc1dd-nkwwq"] Oct 29 23:18:11.006315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:11.006281 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d00a748-d1d3-44a8-b521-eeb79bfde4ed" path="/var/lib/kubelet/pods/3d00a748-d1d3-44a8-b521-eeb79bfde4ed/volumes" Oct 29 23:18:13.643084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.643046 5736 generic.go:356] "Generic (PLEG): container finished" podID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerID="6084ba711f9bf011bd44a2344821a26ab04fccc57ce21888c03772fe26face17" exitCode=0 Oct 29 23:18:13.643475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.643118 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" event={"ID":"7fab7da7-6892-4bd6-976d-7e8bfc3efc13","Type":"ContainerDied","Data":"6084ba711f9bf011bd44a2344821a26ab04fccc57ce21888c03772fe26face17"} Oct 29 23:18:13.645325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.645301 5736 generic.go:356] "Generic (PLEG): container finished" podID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerID="9d5a8165f1f42601dee81bf079b5e98dc871de8d1585edfee9d5802fdfdb70ab" exitCode=0 Oct 29 23:18:13.645497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.645483 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" event={"ID":"277a63e3-aab6-4ed5-a1f8-64ba54797461","Type":"ContainerDied","Data":"9d5a8165f1f42601dee81bf079b5e98dc871de8d1585edfee9d5802fdfdb70ab"} Oct 29 23:18:13.647716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.647655 5736 generic.go:356] "Generic (PLEG): container finished" podID="414097f6-1de2-48ab-add2-979a84a13abf" containerID="7b63381578a9dd6aad401db6b323e7b22e0dfdaaab4f20907edabc3080b0e1ba" exitCode=0 Oct 29 23:18:13.647831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.647726 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" event={"ID":"414097f6-1de2-48ab-add2-979a84a13abf","Type":"ContainerDied","Data":"7b63381578a9dd6aad401db6b323e7b22e0dfdaaab4f20907edabc3080b0e1ba"} Oct 29 23:18:13.782039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.782003 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 23:18:13.832022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.831995 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:18:13.864370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.864340 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:18:13.942711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.942674 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istiod-ca-cert\") pod \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " Oct 29 23:18:13.942948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.942751 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-kube-api-access-gmbvd\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-kube-api-access-gmbvd\") pod \"277a63e3-aab6-4ed5-a1f8-64ba54797461\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " Oct 29 23:18:13.942948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.942810 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-socket\") pod \"277a63e3-aab6-4ed5-a1f8-64ba54797461\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " Oct 29 23:18:13.942948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.942840 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-certs\") pod \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " Oct 29 23:18:13.942948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.942909 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-credential-socket\") pod \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " Oct 29 23:18:13.943187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.942987 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277a63e3-aab6-4ed5-a1f8-64ba54797461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a63e3-aab6-4ed5-a1f8-64ba54797461-istiod-ca-cert\") pod \"277a63e3-aab6-4ed5-a1f8-64ba54797461\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " Oct 29 23:18:13.943187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943009 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-token\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-token\") pod \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " Oct 29 23:18:13.943187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943027 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-podinfo\") pod \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " Oct 29 23:18:13.943187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943083 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-kube-api-access-pwrv6\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-kube-api-access-pwrv6\") pod \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " Oct 29 23:18:13.943187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943142 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-certs" pod "7fab7da7-6892-4bd6-976d-7e8bfc3efc13" (UID: "7fab7da7-6892-4bd6-976d-7e8bfc3efc13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.943187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943148 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-envoy\") pod \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " Oct 29 23:18:13.943506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943211 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-podinfo\") pod \"277a63e3-aab6-4ed5-a1f8-64ba54797461\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " Oct 29 23:18:13.943506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943307 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-data\") pod \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " Oct 29 23:18:13.943506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943335 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-data\") pod \"277a63e3-aab6-4ed5-a1f8-64ba54797461\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " Oct 29 23:18:13.943506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943368 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-socket\") pod \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\" (UID: \"7fab7da7-6892-4bd6-976d-7e8bfc3efc13\") " Oct 29 23:18:13.943506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943385 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-envoy\") pod \"277a63e3-aab6-4ed5-a1f8-64ba54797461\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " Oct 29 23:18:13.943506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943471 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-credential-socket\") pod \"277a63e3-aab6-4ed5-a1f8-64ba54797461\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " Oct 29 23:18:13.943869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943512 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-token\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-token\") pod \"277a63e3-aab6-4ed5-a1f8-64ba54797461\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " Oct 29 23:18:13.943869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943553 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-certs\") pod \"277a63e3-aab6-4ed5-a1f8-64ba54797461\" (UID: \"277a63e3-aab6-4ed5-a1f8-64ba54797461\") " Oct 29 23:18:13.943869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943570 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-credential-socket" pod "7fab7da7-6892-4bd6-976d-7e8bfc3efc13" (UID: "7fab7da7-6892-4bd6-976d-7e8bfc3efc13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.943869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.943818 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-socket" pod "7fab7da7-6892-4bd6-976d-7e8bfc3efc13" (UID: "7fab7da7-6892-4bd6-976d-7e8bfc3efc13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.944550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.944523 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277a63e3-aab6-4ed5-a1f8-64ba54797461-istiod-ca-cert" pod "277a63e3-aab6-4ed5-a1f8-64ba54797461" (UID: "277a63e3-aab6-4ed5-a1f8-64ba54797461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:13.946067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.945368 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-certs" pod "277a63e3-aab6-4ed5-a1f8-64ba54797461" (UID: "277a63e3-aab6-4ed5-a1f8-64ba54797461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.946067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.945422 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-socket" pod "277a63e3-aab6-4ed5-a1f8-64ba54797461" (UID: "277a63e3-aab6-4ed5-a1f8-64ba54797461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.946067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.945677 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-data" pod "7fab7da7-6892-4bd6-976d-7e8bfc3efc13" (UID: "7fab7da7-6892-4bd6-976d-7e8bfc3efc13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.946067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.945613 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istiod-ca-cert" pod "7fab7da7-6892-4bd6-976d-7e8bfc3efc13" (UID: "7fab7da7-6892-4bd6-976d-7e8bfc3efc13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:13.946067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.945873 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-credential-socket" pod "277a63e3-aab6-4ed5-a1f8-64ba54797461" (UID: "277a63e3-aab6-4ed5-a1f8-64ba54797461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.946067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.945885 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-data" pod "277a63e3-aab6-4ed5-a1f8-64ba54797461" (UID: "277a63e3-aab6-4ed5-a1f8-64ba54797461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.946067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.945915 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-podinfo" pod "7fab7da7-6892-4bd6-976d-7e8bfc3efc13" (UID: "7fab7da7-6892-4bd6-976d-7e8bfc3efc13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946441 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-kube-api-access-gmbvd" pod "277a63e3-aab6-4ed5-a1f8-64ba54797461" (UID: "277a63e3-aab6-4ed5-a1f8-64ba54797461"). InnerVolumeSpecName "kube-api-access-gmbvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946497 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946514 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946524 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946533 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946541 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946550 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946560 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a63e3-aab6-4ed5-a1f8-64ba54797461-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946568 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946593 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946606 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.946662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946619 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:13.947169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.946677 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-kube-api-access-pwrv6" pod "7fab7da7-6892-4bd6-976d-7e8bfc3efc13" (UID: "7fab7da7-6892-4bd6-976d-7e8bfc3efc13"). InnerVolumeSpecName "kube-api-access-pwrv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:13.947863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.947833 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-envoy" pod "7fab7da7-6892-4bd6-976d-7e8bfc3efc13" (UID: "7fab7da7-6892-4bd6-976d-7e8bfc3efc13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.947963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.947869 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-podinfo" pod "277a63e3-aab6-4ed5-a1f8-64ba54797461" (UID: "277a63e3-aab6-4ed5-a1f8-64ba54797461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:13.947963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.947932 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-token" pod "7fab7da7-6892-4bd6-976d-7e8bfc3efc13" (UID: "7fab7da7-6892-4bd6-976d-7e8bfc3efc13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:13.948489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.948472 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-envoy" pod "277a63e3-aab6-4ed5-a1f8-64ba54797461" (UID: "277a63e3-aab6-4ed5-a1f8-64ba54797461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:13.949125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:13.949107 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-token" pod "277a63e3-aab6-4ed5-a1f8-64ba54797461" (UID: "277a63e3-aab6-4ed5-a1f8-64ba54797461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:14.047779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.047369 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-certs\") pod \"414097f6-1de2-48ab-add2-979a84a13abf\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " Oct 29 23:18:14.047779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.047469 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/414097f6-1de2-48ab-add2-979a84a13abf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414097f6-1de2-48ab-add2-979a84a13abf-istio-podinfo\") pod \"414097f6-1de2-48ab-add2-979a84a13abf\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " Oct 29 23:18:14.047779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.047620 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-envoy\") pod \"414097f6-1de2-48ab-add2-979a84a13abf\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " Oct 29 23:18:14.047779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.047743 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-kube-api-access-mtkjc\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-kube-api-access-mtkjc\") pod \"414097f6-1de2-48ab-add2-979a84a13abf\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " Oct 29 23:18:14.048493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.047817 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414097f6-1de2-48ab-add2-979a84a13abf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414097f6-1de2-48ab-add2-979a84a13abf-istiod-ca-cert\") pod \"414097f6-1de2-48ab-add2-979a84a13abf\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " Oct 29 23:18:14.048493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.047883 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-credential-socket\") pod \"414097f6-1de2-48ab-add2-979a84a13abf\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " Oct 29 23:18:14.048493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.047946 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-socket\") pod \"414097f6-1de2-48ab-add2-979a84a13abf\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " Oct 29 23:18:14.048493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.048025 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-data\") pod \"414097f6-1de2-48ab-add2-979a84a13abf\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " Oct 29 23:18:14.048493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.048079 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-istio-token\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-istio-token\") pod \"414097f6-1de2-48ab-add2-979a84a13abf\" (UID: \"414097f6-1de2-48ab-add2-979a84a13abf\") " Oct 29 23:18:14.052044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.048432 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-certs" pod "414097f6-1de2-48ab-add2-979a84a13abf" (UID: "414097f6-1de2-48ab-add2-979a84a13abf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:14.052044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.049370 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-socket" pod "414097f6-1de2-48ab-add2-979a84a13abf" (UID: "414097f6-1de2-48ab-add2-979a84a13abf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:14.052336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.049461 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414097f6-1de2-48ab-add2-979a84a13abf-istiod-ca-cert" pod "414097f6-1de2-48ab-add2-979a84a13abf" (UID: "414097f6-1de2-48ab-add2-979a84a13abf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:14.052336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.051167 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-credential-socket" pod "414097f6-1de2-48ab-add2-979a84a13abf" (UID: "414097f6-1de2-48ab-add2-979a84a13abf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:14.052336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.051758 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-data" pod "414097f6-1de2-48ab-add2-979a84a13abf" (UID: "414097f6-1de2-48ab-add2-979a84a13abf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:14.052336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.052004 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/414097f6-1de2-48ab-add2-979a84a13abf-istio-podinfo" pod "414097f6-1de2-48ab-add2-979a84a13abf" (UID: "414097f6-1de2-48ab-add2-979a84a13abf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:14.053268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053244 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053268 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmbvd\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-kube-api-access-gmbvd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053279 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053289 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053298 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053305 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwrv6\" (UniqueName: \"kubernetes.io/projected/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-kube-api-access-pwrv6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053314 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053323 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fab7da7-6892-4bd6-976d-7e8bfc3efc13-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053330 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053339 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414097f6-1de2-48ab-add2-979a84a13abf-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053351 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053364 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a63e3-aab6-4ed5-a1f8-64ba54797461-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.053455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.053376 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414097f6-1de2-48ab-add2-979a84a13abf-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.054267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.054248 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-kube-api-access-mtkjc" pod "414097f6-1de2-48ab-add2-979a84a13abf" (UID: "414097f6-1de2-48ab-add2-979a84a13abf"). InnerVolumeSpecName "kube-api-access-mtkjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:14.054381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.054357 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-istio-token" pod "414097f6-1de2-48ab-add2-979a84a13abf" (UID: "414097f6-1de2-48ab-add2-979a84a13abf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:14.055144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.055123 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-envoy" pod "414097f6-1de2-48ab-add2-979a84a13abf" (UID: "414097f6-1de2-48ab-add2-979a84a13abf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:14.164619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.158245 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtkjc\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-kube-api-access-mtkjc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.164619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.158279 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414097f6-1de2-48ab-add2-979a84a13abf-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.164619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.158295 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414097f6-1de2-48ab-add2-979a84a13abf-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:14.657123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.657091 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" Oct 29 23:18:14.657620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.657114 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5" event={"ID":"414097f6-1de2-48ab-add2-979a84a13abf","Type":"ContainerDied","Data":"51aa2155a9ebf75b9e432e8789dbaebc371e64086c04815c6bac1dbeddc02545"} Oct 29 23:18:14.657620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.657223 5736 scope.go:122] "RemoveContainer" containerID="7b63381578a9dd6aad401db6b323e7b22e0dfdaaab4f20907edabc3080b0e1ba" Oct 29 23:18:14.659738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.659706 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" Oct 29 23:18:14.660431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.660407 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc" event={"ID":"7fab7da7-6892-4bd6-976d-7e8bfc3efc13","Type":"ContainerDied","Data":"d561b74dd824e5fe35bc6e263635e8f9e02408dd938d87e0169eee02c0a3d193"} Oct 29 23:18:14.662128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.662104 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" event={"ID":"277a63e3-aab6-4ed5-a1f8-64ba54797461","Type":"ContainerDied","Data":"8b2996530e94ffe68aae1843f75e2b8ad7468877d690158a8396bca49fa0e76e"} Oct 29 23:18:14.662242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.662200 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr" Oct 29 23:18:14.672134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.672119 5736 scope.go:122] "RemoveContainer" containerID="6084ba711f9bf011bd44a2344821a26ab04fccc57ce21888c03772fe26face17" Oct 29 23:18:14.682787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.682766 5736 scope.go:122] "RemoveContainer" containerID="9d5a8165f1f42601dee81bf079b5e98dc871de8d1585edfee9d5802fdfdb70ab" Oct 29 23:18:14.728495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.728472 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr"] Oct 29 23:18:14.739160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.739073 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-h5ftr"] Oct 29 23:18:14.774231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.774211 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc"] Oct 29 23:18:14.778040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.778012 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-dzlrc"] Oct 29 23:18:14.811621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.811596 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5"] Oct 29 23:18:14.840634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:14.822224 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-hc4m5"] Oct 29 23:18:15.006824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:15.006750 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" path="/var/lib/kubelet/pods/277a63e3-aab6-4ed5-a1f8-64ba54797461/volumes" Oct 29 23:18:15.007278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:15.007263 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414097f6-1de2-48ab-add2-979a84a13abf" path="/var/lib/kubelet/pods/414097f6-1de2-48ab-add2-979a84a13abf/volumes" Oct 29 23:18:15.007746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:15.007727 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" path="/var/lib/kubelet/pods/7fab7da7-6892-4bd6-976d-7e8bfc3efc13/volumes" Oct 29 23:18:20.031433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:20.031394 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq"] Oct 29 23:18:20.042637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:20.042611 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45b45558-qqgbq"] Oct 29 23:18:21.006151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:21.006115 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc96235e-b95e-4849-8507-ed183ad0b87a" path="/var/lib/kubelet/pods/dc96235e-b95e-4849-8507-ed183ad0b87a/volumes" Oct 29 23:18:23.788240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:23.788210 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:18:23.812994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:23.812966 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:18:26.352779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.352748 5736 scope.go:122] "RemoveContainer" containerID="ce9f9a9155587c79a44e61fb30f269222a2481fce29e9913ce19ee45a653aaaf" Oct 29 23:18:26.363129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.363109 5736 scope.go:122] "RemoveContainer" containerID="5173f5474ca753af0995474f2566a65993b9baf42047cfd39c59596fcc8b1a4e" Oct 29 23:18:26.373622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.373599 5736 scope.go:122] "RemoveContainer" containerID="174d20062a9018b7ec1dcd62e13f8c9aacdfa080379ec56c7fd68ddcd93e3dfa" Oct 29 23:18:26.383605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.383574 5736 scope.go:122] "RemoveContainer" containerID="9c89226925216cca4790a0e44d2bf62c70d846bcad3fa19be278c5c9014c0529" Oct 29 23:18:26.393424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.393405 5736 scope.go:122] "RemoveContainer" containerID="295495f47a2ff8e62732b2e84a56a9061111e55a007f27d9f33d2744e3d92382" Oct 29 23:18:26.408010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.407608 5736 scope.go:122] "RemoveContainer" containerID="e702b06d2a4b960aa0874347ee28967a8a74ffda82ac8668a996ff68bc209978" Oct 29 23:18:26.427099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.427073 5736 scope.go:122] "RemoveContainer" containerID="6b4023b64d36913aeccdf606b7a38c7e9fcb9e9f8b5c5b08b5af9ef2282dd58f" Oct 29 23:18:26.439727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.439698 5736 scope.go:122] "RemoveContainer" containerID="a598cf7863fcb9946999fddfe60c10f5b7751335ca3ca948c22f54201e0b94c1" Oct 29 23:18:26.448767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.448741 5736 scope.go:122] "RemoveContainer" containerID="7519e94541f6b6aecb7159799b421d380c0eaa926e7434ae92a5cdf04a61bd3d" Oct 29 23:18:26.457530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.457512 5736 scope.go:122] "RemoveContainer" containerID="790b20112870ae62ce27ebf24557232a0b1f352cec22a1c2066bfab0b9268275" Oct 29 23:18:26.466643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:26.466622 5736 scope.go:122] "RemoveContainer" containerID="7f90132aacacb74a1812587e201efc76ff0b5257e99d43c6afe4a6e4352e6d39" Oct 29 23:18:26.825983 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:18:26.825947 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:18:30.613755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.613719 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm"] Oct 29 23:18:30.614443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614426 5736 cpu_manager.go:414] "containerMap: removing container" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" Oct 29 23:18:30.614443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614444 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" Oct 29 23:18:30.614546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614457 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" Oct 29 23:18:30.614546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614462 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" Oct 29 23:18:30.614546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614495 5736 cpu_manager.go:414] "containerMap: removing container" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" Oct 29 23:18:30.614546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614504 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" Oct 29 23:18:30.614546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614525 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7ed5370d-2a2d-44b0-9ef0-1d4d683fe447" containerName="revoke-keys" Oct 29 23:18:30.614546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614530 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed5370d-2a2d-44b0-9ef0-1d4d683fe447" containerName="revoke-keys" Oct 29 23:18:30.614768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614668 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed5370d-2a2d-44b0-9ef0-1d4d683fe447" containerName="revoke-keys" Oct 29 23:18:30.614768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614686 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="414097f6-1de2-48ab-add2-979a84a13abf" containerName="istio-proxy" Oct 29 23:18:30.614768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614693 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fab7da7-6892-4bd6-976d-7e8bfc3efc13" containerName="istio-proxy" Oct 29 23:18:30.614768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.614702 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="277a63e3-aab6-4ed5-a1f8-64ba54797461" containerName="istio-proxy" Oct 29 23:18:30.620692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.620673 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.636737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.636717 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:18:30.655047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.655018 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm"] Oct 29 23:18:30.668234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.668204 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7wf6\" (UniqueName: \"kubernetes.io/projected/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-kube-api-access-k7wf6\") pod \"maas-api-revoke-sub-26c8ddf8-v74jm\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.668532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.668514 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c8ddf8-v74jm\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.668671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.668656 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c8ddf8-v74jm\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.737233 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.737197 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:18:30.770435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.770403 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c8ddf8-v74jm\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.770662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.770502 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c8ddf8-v74jm\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.770662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.770630 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7wf6\" (UniqueName: \"kubernetes.io/projected/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-kube-api-access-k7wf6\") pod \"maas-api-revoke-sub-26c8ddf8-v74jm\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.771336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.771307 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c8ddf8-v74jm\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.772948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.772925 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c8ddf8-v74jm\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.794223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.794200 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7wf6\" (UniqueName: \"kubernetes.io/projected/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-kube-api-access-k7wf6\") pod \"maas-api-revoke-sub-26c8ddf8-v74jm\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:30.935531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:30.935457 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:31.245939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:31.245911 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm"] Oct 29 23:18:31.249036 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:18:31.249007 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f82ec59_ff49_45ed_ad06_c7c88ced15f0.slice/crio-15dbd53770848a00735b87c1751add7f2d82dfd580a0778209f5ac9628e7e9a9 WatchSource:0}: Error finding container 15dbd53770848a00735b87c1751add7f2d82dfd580a0778209f5ac9628e7e9a9: Status 404 returned error can't find the container with id 15dbd53770848a00735b87c1751add7f2d82dfd580a0778209f5ac9628e7e9a9 Oct 29 23:18:31.827664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:31.827631 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" event={"ID":"5f82ec59-ff49-45ed-ad06-c7c88ced15f0","Type":"ContainerStarted","Data":"eeb1abe12fa3e31af2392681cb79f4eb545d2c9bf5bf627832ed727e21db233a"} Oct 29 23:18:31.828077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:31.827682 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" event={"ID":"5f82ec59-ff49-45ed-ad06-c7c88ced15f0","Type":"ContainerStarted","Data":"15dbd53770848a00735b87c1751add7f2d82dfd580a0778209f5ac9628e7e9a9"} Oct 29 23:18:31.870391 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:31.870343 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" podStartSLOduration=1.8703289760000001 podStartE2EDuration="1.870328976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:31.866158138 +0000 UTC m=+2108.982077744" watchObservedRunningTime="2026-10-29 23:18:31.870328976 +0000 UTC m=+2108.986248581" Oct 29 23:18:32.837276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:32.837244 5736 generic.go:356] "Generic (PLEG): container finished" podID="5f82ec59-ff49-45ed-ad06-c7c88ced15f0" containerID="eeb1abe12fa3e31af2392681cb79f4eb545d2c9bf5bf627832ed727e21db233a" exitCode=0 Oct 29 23:18:32.837741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:32.837328 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" event={"ID":"5f82ec59-ff49-45ed-ad06-c7c88ced15f0","Type":"ContainerDied","Data":"eeb1abe12fa3e31af2392681cb79f4eb545d2c9bf5bf627832ed727e21db233a"} Oct 29 23:18:34.021832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.021805 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:34.137349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.137259 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-openshift-ca\") pod \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " Oct 29 23:18:34.137349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.137324 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-xks-ca\") pod \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " Oct 29 23:18:34.137631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.137425 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-kube-api-access-k7wf6\" (UniqueName: \"kubernetes.io/projected/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-kube-api-access-k7wf6\") pod \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\" (UID: \"5f82ec59-ff49-45ed-ad06-c7c88ced15f0\") " Oct 29 23:18:34.137713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.137658 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-openshift-ca" pod "5f82ec59-ff49-45ed-ad06-c7c88ced15f0" (UID: "5f82ec59-ff49-45ed-ad06-c7c88ced15f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:34.138106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.138085 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:34.139540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.139514 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-kube-api-access-k7wf6" pod "5f82ec59-ff49-45ed-ad06-c7c88ced15f0" (UID: "5f82ec59-ff49-45ed-ad06-c7c88ced15f0"). InnerVolumeSpecName "kube-api-access-k7wf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:34.139714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.139545 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-xks-ca" pod "5f82ec59-ff49-45ed-ad06-c7c88ced15f0" (UID: "5f82ec59-ff49-45ed-ad06-c7c88ced15f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:34.240147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.240118 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:34.240147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.240147 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7wf6\" (UniqueName: \"kubernetes.io/projected/5f82ec59-ff49-45ed-ad06-c7c88ced15f0-kube-api-access-k7wf6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:34.852808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.852780 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" Oct 29 23:18:34.852994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.852817 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm" event={"ID":"5f82ec59-ff49-45ed-ad06-c7c88ced15f0","Type":"ContainerDied","Data":"15dbd53770848a00735b87c1751add7f2d82dfd580a0778209f5ac9628e7e9a9"} Oct 29 23:18:34.852994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:34.852887 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15dbd53770848a00735b87c1751add7f2d82dfd580a0778209f5ac9628e7e9a9" Oct 29 23:18:37.006925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:37.006897 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r"] Oct 29 23:18:37.007352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:37.007067 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" podUID="138b5b65-38aa-4c67-916f-1047e5898300" containerName="istio-proxy" containerID="cri-o://ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60" gracePeriod=30 Oct 29 23:18:40.676358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:40.676323 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85"] Oct 29 23:18:40.676886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:40.676517 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" containerID="cri-o://92ea1a1f028c0140728146266dcd6ad2cb5164b6de819e76e2c3260bde8ca9b2" gracePeriod=30 Oct 29 23:18:42.290980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.290952 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:18:42.385250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385212 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-credential-socket\") pod \"138b5b65-38aa-4c67-916f-1047e5898300\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " Oct 29 23:18:42.385411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385282 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-data\") pod \"138b5b65-38aa-4c67-916f-1047e5898300\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " Oct 29 23:18:42.385465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385422 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-socket\") pod \"138b5b65-38aa-4c67-916f-1047e5898300\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " Oct 29 23:18:42.385558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385531 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-istio-token\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-istio-token\") pod \"138b5b65-38aa-4c67-916f-1047e5898300\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " Oct 29 23:18:42.385653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385527 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-credential-socket" pod "138b5b65-38aa-4c67-916f-1047e5898300" (UID: "138b5b65-38aa-4c67-916f-1047e5898300"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:42.385653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385574 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-certs\") pod \"138b5b65-38aa-4c67-916f-1047e5898300\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " Oct 29 23:18:42.385653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385640 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-data" pod "138b5b65-38aa-4c67-916f-1047e5898300" (UID: "138b5b65-38aa-4c67-916f-1047e5898300"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:42.385869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385738 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/138b5b65-38aa-4c67-916f-1047e5898300-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138b5b65-38aa-4c67-916f-1047e5898300-istio-podinfo\") pod \"138b5b65-38aa-4c67-916f-1047e5898300\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " Oct 29 23:18:42.385869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385766 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-kube-api-access-sfxfd\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-kube-api-access-sfxfd\") pod \"138b5b65-38aa-4c67-916f-1047e5898300\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " Oct 29 23:18:42.385869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385825 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-envoy\") pod \"138b5b65-38aa-4c67-916f-1047e5898300\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " Oct 29 23:18:42.385869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385828 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-socket" pod "138b5b65-38aa-4c67-916f-1047e5898300" (UID: "138b5b65-38aa-4c67-916f-1047e5898300"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:42.385869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385859 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138b5b65-38aa-4c67-916f-1047e5898300-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138b5b65-38aa-4c67-916f-1047e5898300-istiod-ca-cert\") pod \"138b5b65-38aa-4c67-916f-1047e5898300\" (UID: \"138b5b65-38aa-4c67-916f-1047e5898300\") " Oct 29 23:18:42.386170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.385984 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-certs" pod "138b5b65-38aa-4c67-916f-1047e5898300" (UID: "138b5b65-38aa-4c67-916f-1047e5898300"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:42.386406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.386344 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138b5b65-38aa-4c67-916f-1047e5898300-istiod-ca-cert" pod "138b5b65-38aa-4c67-916f-1047e5898300" (UID: "138b5b65-38aa-4c67-916f-1047e5898300"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:42.386660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.386640 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138b5b65-38aa-4c67-916f-1047e5898300-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:42.386753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.386665 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:42.386753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.386679 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:42.386753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.386692 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:42.386753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.386706 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:42.388472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.388449 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/138b5b65-38aa-4c67-916f-1047e5898300-istio-podinfo" pod "138b5b65-38aa-4c67-916f-1047e5898300" (UID: "138b5b65-38aa-4c67-916f-1047e5898300"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:42.388550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.388503 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-envoy" pod "138b5b65-38aa-4c67-916f-1047e5898300" (UID: "138b5b65-38aa-4c67-916f-1047e5898300"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:42.389024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.388996 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-kube-api-access-sfxfd" pod "138b5b65-38aa-4c67-916f-1047e5898300" (UID: "138b5b65-38aa-4c67-916f-1047e5898300"). InnerVolumeSpecName "kube-api-access-sfxfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:42.412162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.406046 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-istio-token" pod "138b5b65-38aa-4c67-916f-1047e5898300" (UID: "138b5b65-38aa-4c67-916f-1047e5898300"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:42.439529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.439496 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv"] Oct 29 23:18:42.439804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.439750 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" containerID="cri-o://15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15" gracePeriod=30 Oct 29 23:18:42.491509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.491472 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138b5b65-38aa-4c67-916f-1047e5898300-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:42.491509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.491511 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfxfd\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-kube-api-access-sfxfd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:42.491741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.491528 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138b5b65-38aa-4c67-916f-1047e5898300-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:42.491741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.491542 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138b5b65-38aa-4c67-916f-1047e5898300-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:42.932423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.926692 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" Oct 29 23:18:42.932423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.926573 5736 generic.go:356] "Generic (PLEG): container finished" podID="138b5b65-38aa-4c67-916f-1047e5898300" containerID="ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60" exitCode=0 Oct 29 23:18:42.932423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.926766 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" event={"ID":"138b5b65-38aa-4c67-916f-1047e5898300","Type":"ContainerDied","Data":"ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60"} Oct 29 23:18:42.932423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.926844 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r" event={"ID":"138b5b65-38aa-4c67-916f-1047e5898300","Type":"ContainerDied","Data":"47831ffbce52db2fe92c54eaff38a5f39fba5868e4022824a27c5c98a2c7d5cf"} Oct 29 23:18:42.932423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.926885 5736 scope.go:122] "RemoveContainer" containerID="ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60" Oct 29 23:18:42.940891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.940870 5736 scope.go:122] "RemoveContainer" containerID="ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60" Oct 29 23:18:42.941207 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:18:42.941176 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60\": container with ID starting with ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60 not found: ID does not exist" containerID="ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60" Oct 29 23:18:42.941314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.941215 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60"} err="failed to get container status \"ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60\": rpc error: code = NotFound desc = could not find container \"ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60\": container with ID starting with ef62247f90ba9ab523bcc50fc1232fffda62f2e558d9a19cd69f13177f03cb60 not found: ID does not exist" Oct 29 23:18:42.981139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.981106 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r"] Oct 29 23:18:42.992696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:42.992671 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-r297r"] Oct 29 23:18:43.005649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:43.005626 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138b5b65-38aa-4c67-916f-1047e5898300" path="/var/lib/kubelet/pods/138b5b65-38aa-4c67-916f-1047e5898300/volumes" Oct 29 23:18:45.952982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:45.952913 5736 generic.go:356] "Generic (PLEG): container finished" podID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerID="92ea1a1f028c0140728146266dcd6ad2cb5164b6de819e76e2c3260bde8ca9b2" exitCode=0 Oct 29 23:18:45.953416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:45.952986 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" event={"ID":"1abd5efc-ca12-4692-bc81-e62dadfd60a7","Type":"ContainerDied","Data":"92ea1a1f028c0140728146266dcd6ad2cb5164b6de819e76e2c3260bde8ca9b2"} Oct 29 23:18:45.953416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:45.953052 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" event={"ID":"1abd5efc-ca12-4692-bc81-e62dadfd60a7","Type":"ContainerDied","Data":"4fc7fd361157283c8df7462cada6dc6716a0b57b381cb6756933c389b572ab96"} Oct 29 23:18:45.953416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:45.953075 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fc7fd361157283c8df7462cada6dc6716a0b57b381cb6756933c389b572ab96" Oct 29 23:18:45.956316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:45.956296 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.064720 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-envoy\") pod \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.064838 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-token\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-token\") pod \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.064916 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-credential-socket\") pod \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.064963 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-certs\") pod \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.065041 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-socket\") pod \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.065290 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-podinfo\") pod \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.065360 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-data\") pod \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.065413 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-kube-api-access-q2pr2\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-kube-api-access-q2pr2\") pod \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.065475 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-credential-socket" pod "1abd5efc-ca12-4692-bc81-e62dadfd60a7" (UID: "1abd5efc-ca12-4692-bc81-e62dadfd60a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:46.065736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.065500 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istiod-ca-cert\") pod \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\" (UID: \"1abd5efc-ca12-4692-bc81-e62dadfd60a7\") " Oct 29 23:18:46.066380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.066285 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:46.066781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.066755 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-certs" pod "1abd5efc-ca12-4692-bc81-e62dadfd60a7" (UID: "1abd5efc-ca12-4692-bc81-e62dadfd60a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:46.066942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.066884 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-socket" pod "1abd5efc-ca12-4692-bc81-e62dadfd60a7" (UID: "1abd5efc-ca12-4692-bc81-e62dadfd60a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:46.067717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.067690 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-token" pod "1abd5efc-ca12-4692-bc81-e62dadfd60a7" (UID: "1abd5efc-ca12-4692-bc81-e62dadfd60a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:46.068056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.068037 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-data" pod "1abd5efc-ca12-4692-bc81-e62dadfd60a7" (UID: "1abd5efc-ca12-4692-bc81-e62dadfd60a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:46.068785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.068760 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-podinfo" pod "1abd5efc-ca12-4692-bc81-e62dadfd60a7" (UID: "1abd5efc-ca12-4692-bc81-e62dadfd60a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:46.069495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.069473 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istiod-ca-cert" pod "1abd5efc-ca12-4692-bc81-e62dadfd60a7" (UID: "1abd5efc-ca12-4692-bc81-e62dadfd60a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:46.069943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.069914 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-kube-api-access-q2pr2" pod "1abd5efc-ca12-4692-bc81-e62dadfd60a7" (UID: "1abd5efc-ca12-4692-bc81-e62dadfd60a7"). InnerVolumeSpecName "kube-api-access-q2pr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:46.082999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.082976 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-envoy" pod "1abd5efc-ca12-4692-bc81-e62dadfd60a7" (UID: "1abd5efc-ca12-4692-bc81-e62dadfd60a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:46.182205 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.176365 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2pr2\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-kube-api-access-q2pr2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:46.182532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.182491 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:46.182819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.182801 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:46.182962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.182946 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:46.183061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.183050 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:46.183153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.183142 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:46.183301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.183288 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:46.185634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.185611 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abd5efc-ca12-4692-bc81-e62dadfd60a7-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:46.960003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:46.959969 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85" Oct 29 23:18:47.025530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.025497 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85"] Oct 29 23:18:47.037210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.037185 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-5dm85"] Oct 29 23:18:47.719945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.719919 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:18:47.910953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.910683 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-kube-api-access-xnf6c\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-kube-api-access-xnf6c\") pod \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " Oct 29 23:18:47.910953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.910768 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-data\") pod \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " Oct 29 23:18:47.910953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.910804 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-socket\") pod \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " Oct 29 23:18:47.910953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.910888 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-podinfo\") pod \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " Oct 29 23:18:47.910953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.910960 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-token\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-token\") pod \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " Oct 29 23:18:47.911442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.911143 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-credential-socket\") pod \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " Oct 29 23:18:47.911442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.911261 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-envoy\") pod \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " Oct 29 23:18:47.911442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.911292 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istiod-ca-cert\") pod \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " Oct 29 23:18:47.940968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.939746 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-podinfo" pod "06bbb4ac-5639-4e05-8260-6ef27d0e526c" (UID: "06bbb4ac-5639-4e05-8260-6ef27d0e526c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:47.941567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.941530 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-token" pod "06bbb4ac-5639-4e05-8260-6ef27d0e526c" (UID: "06bbb4ac-5639-4e05-8260-6ef27d0e526c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:47.941839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.941808 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-credential-socket" pod "06bbb4ac-5639-4e05-8260-6ef27d0e526c" (UID: "06bbb4ac-5639-4e05-8260-6ef27d0e526c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:47.944728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.944303 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istiod-ca-cert" pod "06bbb4ac-5639-4e05-8260-6ef27d0e526c" (UID: "06bbb4ac-5639-4e05-8260-6ef27d0e526c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:47.945053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.944886 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-data" pod "06bbb4ac-5639-4e05-8260-6ef27d0e526c" (UID: "06bbb4ac-5639-4e05-8260-6ef27d0e526c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:47.946849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.945175 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-socket" pod "06bbb4ac-5639-4e05-8260-6ef27d0e526c" (UID: "06bbb4ac-5639-4e05-8260-6ef27d0e526c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:47.946849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.946811 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-kube-api-access-xnf6c" pod "06bbb4ac-5639-4e05-8260-6ef27d0e526c" (UID: "06bbb4ac-5639-4e05-8260-6ef27d0e526c"). InnerVolumeSpecName "kube-api-access-xnf6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:47.947036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.946903 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-certs\") pod \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\" (UID: \"06bbb4ac-5639-4e05-8260-6ef27d0e526c\") " Oct 29 23:18:47.947464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.947423 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-certs" pod "06bbb4ac-5639-4e05-8260-6ef27d0e526c" (UID: "06bbb4ac-5639-4e05-8260-6ef27d0e526c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:47.947927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.947868 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:47.947927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.947897 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnf6c\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-kube-api-access-xnf6c\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:47.947927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.947909 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:47.947927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.947921 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:47.948332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.947934 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:47.948332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.947945 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:47.948332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.947956 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:47.948332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.947968 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:47.959672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.949950 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-envoy" pod "06bbb4ac-5639-4e05-8260-6ef27d0e526c" (UID: "06bbb4ac-5639-4e05-8260-6ef27d0e526c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:47.976730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.976634 5736 generic.go:356] "Generic (PLEG): container finished" podID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerID="15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15" exitCode=0 Oct 29 23:18:47.976730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.976713 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" Oct 29 23:18:47.977191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.976734 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" event={"ID":"06bbb4ac-5639-4e05-8260-6ef27d0e526c","Type":"ContainerDied","Data":"15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15"} Oct 29 23:18:47.977191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.976805 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv" event={"ID":"06bbb4ac-5639-4e05-8260-6ef27d0e526c","Type":"ContainerDied","Data":"f76b9eecf0f8f4af533ab708319b10017a3cf030f0edfb871d6db904c26980af"} Oct 29 23:18:47.977191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.976837 5736 scope.go:122] "RemoveContainer" containerID="15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15" Oct 29 23:18:47.989693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.989672 5736 scope.go:122] "RemoveContainer" containerID="15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15" Oct 29 23:18:47.989983 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:18:47.989960 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15\": container with ID starting with 15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15 not found: ID does not exist" containerID="15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15" Oct 29 23:18:47.990073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:47.989991 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15"} err="failed to get container status \"15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15\": rpc error: code = NotFound desc = could not find container \"15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15\": container with ID starting with 15315c7b7beb82d1c40ab5b24ad18abe69646250088b58e2bea43de8d537bc15 not found: ID does not exist" Oct 29 23:18:48.033979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:48.033947 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv"] Oct 29 23:18:48.037514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:48.037478 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-v5tmv"] Oct 29 23:18:48.049805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:48.049777 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bbb4ac-5639-4e05-8260-6ef27d0e526c-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:49.006477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:49.006443 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" path="/var/lib/kubelet/pods/06bbb4ac-5639-4e05-8260-6ef27d0e526c/volumes" Oct 29 23:18:49.007031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:49.007013 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" path="/var/lib/kubelet/pods/1abd5efc-ca12-4692-bc81-e62dadfd60a7/volumes" Oct 29 23:18:49.674170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:49.674135 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj"] Oct 29 23:18:49.674410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:49.674385 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" containerID="cri-o://7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df" gracePeriod=30 Oct 29 23:18:54.962192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:54.962165 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:18:55.055280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.055237 5736 generic.go:356] "Generic (PLEG): container finished" podID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerID="7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df" exitCode=0 Oct 29 23:18:55.055489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.055367 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" event={"ID":"0d85aebb-8ada-4414-8663-4af63145e5ce","Type":"ContainerDied","Data":"7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df"} Oct 29 23:18:55.055489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.055446 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" event={"ID":"0d85aebb-8ada-4414-8663-4af63145e5ce","Type":"ContainerDied","Data":"9e4c646f1eabfc16ea07e94d7af76a4a5ad2caf8f3199f1b51e5dab89b48cc7b"} Oct 29 23:18:55.055489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.055475 5736 scope.go:122] "RemoveContainer" containerID="7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df" Oct 29 23:18:55.055719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.055671 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj" Oct 29 23:18:55.071362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.071344 5736 scope.go:122] "RemoveContainer" containerID="7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df" Oct 29 23:18:55.071665 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:18:55.071643 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df\": container with ID starting with 7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df not found: ID does not exist" containerID="7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df" Oct 29 23:18:55.071752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.071677 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df"} err="failed to get container status \"7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df\": rpc error: code = NotFound desc = could not find container \"7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df\": container with ID starting with 7d86745726f5e2ccedbd88a3149446cbc42b33e02c01b75e75f4fde9bf81e5df not found: ID does not exist" Oct 29 23:18:55.096518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.096488 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-podinfo\") pod \"0d85aebb-8ada-4414-8663-4af63145e5ce\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " Oct 29 23:18:55.096669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.096548 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-kube-api-access-p4j78\") pod \"0d85aebb-8ada-4414-8663-4af63145e5ce\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " Oct 29 23:18:55.096669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.096594 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-certs\") pod \"0d85aebb-8ada-4414-8663-4af63145e5ce\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " Oct 29 23:18:55.096806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.096729 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-envoy\") pod \"0d85aebb-8ada-4414-8663-4af63145e5ce\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " Oct 29 23:18:55.096806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.096799 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-credential-socket\") pod \"0d85aebb-8ada-4414-8663-4af63145e5ce\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " Oct 29 23:18:55.096920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.096843 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d85aebb-8ada-4414-8663-4af63145e5ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d85aebb-8ada-4414-8663-4af63145e5ce-istiod-ca-cert\") pod \"0d85aebb-8ada-4414-8663-4af63145e5ce\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " Oct 29 23:18:55.096983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.096955 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-token\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-token\") pod \"0d85aebb-8ada-4414-8663-4af63145e5ce\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " Oct 29 23:18:55.097096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.097074 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-data\") pod \"0d85aebb-8ada-4414-8663-4af63145e5ce\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " Oct 29 23:18:55.097194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.097094 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-certs" pod "0d85aebb-8ada-4414-8663-4af63145e5ce" (UID: "0d85aebb-8ada-4414-8663-4af63145e5ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.097257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.097195 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-credential-socket" pod "0d85aebb-8ada-4414-8663-4af63145e5ce" (UID: "0d85aebb-8ada-4414-8663-4af63145e5ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.097257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.097204 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-socket\") pod \"0d85aebb-8ada-4414-8663-4af63145e5ce\" (UID: \"0d85aebb-8ada-4414-8663-4af63145e5ce\") " Oct 29 23:18:55.097381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.097304 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d85aebb-8ada-4414-8663-4af63145e5ce-istiod-ca-cert" pod "0d85aebb-8ada-4414-8663-4af63145e5ce" (UID: "0d85aebb-8ada-4414-8663-4af63145e5ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:55.097971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.097616 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-data" pod "0d85aebb-8ada-4414-8663-4af63145e5ce" (UID: "0d85aebb-8ada-4414-8663-4af63145e5ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.097971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.097748 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-socket" pod "0d85aebb-8ada-4414-8663-4af63145e5ce" (UID: "0d85aebb-8ada-4414-8663-4af63145e5ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.098475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.098296 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.098475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.098322 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d85aebb-8ada-4414-8663-4af63145e5ce-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.098475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.098337 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.098475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.098350 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.098475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.098363 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.099367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.099005 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-kube-api-access-p4j78" pod "0d85aebb-8ada-4414-8663-4af63145e5ce" (UID: "0d85aebb-8ada-4414-8663-4af63145e5ce"). InnerVolumeSpecName "kube-api-access-p4j78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.099367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.099142 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-envoy" pod "0d85aebb-8ada-4414-8663-4af63145e5ce" (UID: "0d85aebb-8ada-4414-8663-4af63145e5ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:55.099367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.099145 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-podinfo" pod "0d85aebb-8ada-4414-8663-4af63145e5ce" (UID: "0d85aebb-8ada-4414-8663-4af63145e5ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:55.099999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.099976 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-token" pod "0d85aebb-8ada-4414-8663-4af63145e5ce" (UID: "0d85aebb-8ada-4414-8663-4af63145e5ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:55.222665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.219498 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.222665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.221619 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4j78\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-kube-api-access-p4j78\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.222665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.221647 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.222665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.221664 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d85aebb-8ada-4414-8663-4af63145e5ce-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:18:55.415600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.411662 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj"] Oct 29 23:18:55.415600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:55.415473 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-xd8gj"] Oct 29 23:18:56.824231 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:18:56.824189 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:18:57.010067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:18:57.010034 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" path="/var/lib/kubelet/pods/0d85aebb-8ada-4414-8663-4af63145e5ce/volumes" Oct 29 23:19:03.242364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:03.242332 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx"] Oct 29 23:19:03.242853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:03.242536 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" containerID="cri-o://c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c" gracePeriod=30 Oct 29 23:19:04.931512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:04.931474 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:19:04.931990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:04.931540 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:19:08.745675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.745642 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 23:19:08.835606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.834149 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-token\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-token\") pod \"de92d227-c2af-4a2c-92bc-6c85b4e77056\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " Oct 29 23:19:08.835606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.834217 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de92d227-c2af-4a2c-92bc-6c85b4e77056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de92d227-c2af-4a2c-92bc-6c85b4e77056-istiod-ca-cert\") pod \"de92d227-c2af-4a2c-92bc-6c85b4e77056\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " Oct 29 23:19:08.835606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.834351 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-podinfo\") pod \"de92d227-c2af-4a2c-92bc-6c85b4e77056\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " Oct 29 23:19:08.835606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.834387 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-certs\") pod \"de92d227-c2af-4a2c-92bc-6c85b4e77056\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " Oct 29 23:19:08.835606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.834467 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-data\") pod \"de92d227-c2af-4a2c-92bc-6c85b4e77056\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " Oct 29 23:19:08.835606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.834599 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-credential-socket\") pod \"de92d227-c2af-4a2c-92bc-6c85b4e77056\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " Oct 29 23:19:08.835606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.834651 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-kube-api-access-d2slc\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-kube-api-access-d2slc\") pod \"de92d227-c2af-4a2c-92bc-6c85b4e77056\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " Oct 29 23:19:08.835606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.834724 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-socket\") pod \"de92d227-c2af-4a2c-92bc-6c85b4e77056\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " Oct 29 23:19:08.835606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.834886 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-envoy\") pod \"de92d227-c2af-4a2c-92bc-6c85b4e77056\" (UID: \"de92d227-c2af-4a2c-92bc-6c85b4e77056\") " Oct 29 23:19:08.840673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.836859 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de92d227-c2af-4a2c-92bc-6c85b4e77056-istiod-ca-cert" pod "de92d227-c2af-4a2c-92bc-6c85b4e77056" (UID: "de92d227-c2af-4a2c-92bc-6c85b4e77056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:08.840673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.837181 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-certs" pod "de92d227-c2af-4a2c-92bc-6c85b4e77056" (UID: "de92d227-c2af-4a2c-92bc-6c85b4e77056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:08.841510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.841486 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-data" pod "de92d227-c2af-4a2c-92bc-6c85b4e77056" (UID: "de92d227-c2af-4a2c-92bc-6c85b4e77056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:08.841869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.841849 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-credential-socket" pod "de92d227-c2af-4a2c-92bc-6c85b4e77056" (UID: "de92d227-c2af-4a2c-92bc-6c85b4e77056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:08.853775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.853730 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-socket" pod "de92d227-c2af-4a2c-92bc-6c85b4e77056" (UID: "de92d227-c2af-4a2c-92bc-6c85b4e77056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:08.869493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.866768 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-kube-api-access-d2slc" pod "de92d227-c2af-4a2c-92bc-6c85b4e77056" (UID: "de92d227-c2af-4a2c-92bc-6c85b4e77056"). InnerVolumeSpecName "kube-api-access-d2slc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:08.884290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.884119 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-token" pod "de92d227-c2af-4a2c-92bc-6c85b4e77056" (UID: "de92d227-c2af-4a2c-92bc-6c85b4e77056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:08.884290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.884276 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-envoy" pod "de92d227-c2af-4a2c-92bc-6c85b4e77056" (UID: "de92d227-c2af-4a2c-92bc-6c85b4e77056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:08.932809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.917891 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-podinfo" pod "de92d227-c2af-4a2c-92bc-6c85b4e77056" (UID: "de92d227-c2af-4a2c-92bc-6c85b4e77056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:08.940216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.940060 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:08.940216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.940092 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:08.940216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.940106 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de92d227-c2af-4a2c-92bc-6c85b4e77056-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:08.940216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.940121 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:08.940216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.940135 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:08.940216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.940150 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:08.940216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.940164 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:08.940216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.940177 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2slc\" (UniqueName: \"kubernetes.io/projected/de92d227-c2af-4a2c-92bc-6c85b4e77056-kube-api-access-d2slc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:08.940216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:08.940192 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de92d227-c2af-4a2c-92bc-6c85b4e77056-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:09.168613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:09.167073 5736 generic.go:356] "Generic (PLEG): container finished" podID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerID="c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c" exitCode=0 Oct 29 23:19:09.168613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:09.167151 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" event={"ID":"de92d227-c2af-4a2c-92bc-6c85b4e77056","Type":"ContainerDied","Data":"c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c"} Oct 29 23:19:09.168613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:09.167209 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" event={"ID":"de92d227-c2af-4a2c-92bc-6c85b4e77056","Type":"ContainerDied","Data":"8122e2daf74b4d082d477f1cbe33e81eaa82c7ee23698e3831b7534adf5dde34"} Oct 29 23:19:09.168613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:09.167237 5736 scope.go:122] "RemoveContainer" containerID="c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c" Oct 29 23:19:09.168613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:09.168094 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx" Oct 29 23:19:09.181487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:09.181463 5736 scope.go:122] "RemoveContainer" containerID="c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c" Oct 29 23:19:09.181884 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:19:09.181847 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c\": container with ID starting with c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c not found: ID does not exist" containerID="c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c" Oct 29 23:19:09.182030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:09.181883 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c"} err="failed to get container status \"c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c\": rpc error: code = NotFound desc = could not find container \"c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c\": container with ID starting with c198ea18bc16a933e0a6a5ada7ba8d4cfaec5d0dd0bb5e88b87bb596248ef77c not found: ID does not exist" Oct 29 23:19:09.227133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:09.227092 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx"] Oct 29 23:19:09.238779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:09.238738 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ds5sx"] Oct 29 23:19:11.009744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:11.009706 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" path="/var/lib/kubelet/pods/de92d227-c2af-4a2c-92bc-6c85b4e77056/volumes" Oct 29 23:19:19.430438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:19.429892 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s"] Oct 29 23:19:19.436598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:19.436544 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96cd49-m928s"] Oct 29 23:19:21.005454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:21.005420 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed5370d-2a2d-44b0-9ef0-1d4d683fe447" path="/var/lib/kubelet/pods/7ed5370d-2a2d-44b0-9ef0-1d4d683fe447/volumes" Oct 29 23:19:24.865116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:24.865077 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk"] Oct 29 23:19:26.565739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:26.565708 5736 scope.go:122] "RemoveContainer" containerID="4b28c8347d07a660c1e72fc4ac20b6aa58d7e9575092daaa482f4062349dfaf1" Oct 29 23:19:26.576032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:26.576014 5736 scope.go:122] "RemoveContainer" containerID="92ea1a1f028c0140728146266dcd6ad2cb5164b6de819e76e2c3260bde8ca9b2" Oct 29 23:19:26.586099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:26.586076 5736 scope.go:122] "RemoveContainer" containerID="903a756f6c825084c975f1840bd4ebefd2b8d1dc6296bd25f9befcf8dca56c67" Oct 29 23:19:26.825692 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:19:26.825560 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:19:28.842892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:28.842828 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw"] Oct 29 23:19:28.843320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:28.843078 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" containerID="cri-o://49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139" gracePeriod=30 Oct 29 23:19:28.853146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:28.853119 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2"] Oct 29 23:19:28.853362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:28.853339 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" containerID="cri-o://5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e" gracePeriod=30 Oct 29 23:19:28.857352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:28.857323 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r"] Oct 29 23:19:28.857572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:28.857528 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" containerID="cri-o://7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a" gracePeriod=30 Oct 29 23:19:34.179448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.179153 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:19:34.185786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.185761 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:19:34.190754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.190731 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 23:19:34.202248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202223 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istiod-ca-cert\") pod \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " Oct 29 23:19:34.202371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202264 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-kube-api-access-24f2v\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-kube-api-access-24f2v\") pod \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " Oct 29 23:19:34.202371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202305 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-token\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-token\") pod \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " Oct 29 23:19:34.202496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202414 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-credential-socket\") pod \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " Oct 29 23:19:34.202496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202470 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-socket\") pod \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " Oct 29 23:19:34.202630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202560 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-data\") pod \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " Oct 29 23:19:34.202695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202638 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-certs\") pod \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " Oct 29 23:19:34.202695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202681 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istiod-ca-cert" pod "85cd456c-fb72-48b1-91e5-c46d41e8dec3" (UID: "85cd456c-fb72-48b1-91e5-c46d41e8dec3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:34.202810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202705 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-envoy\") pod \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " Oct 29 23:19:34.202810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202708 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-credential-socket" pod "85cd456c-fb72-48b1-91e5-c46d41e8dec3" (UID: "85cd456c-fb72-48b1-91e5-c46d41e8dec3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.202918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202799 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-socket" pod "85cd456c-fb72-48b1-91e5-c46d41e8dec3" (UID: "85cd456c-fb72-48b1-91e5-c46d41e8dec3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.202918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202819 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-podinfo\") pod \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\" (UID: \"85cd456c-fb72-48b1-91e5-c46d41e8dec3\") " Oct 29 23:19:34.202918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202908 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-certs" pod "85cd456c-fb72-48b1-91e5-c46d41e8dec3" (UID: "85cd456c-fb72-48b1-91e5-c46d41e8dec3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.203083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.202984 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-data" pod "85cd456c-fb72-48b1-91e5-c46d41e8dec3" (UID: "85cd456c-fb72-48b1-91e5-c46d41e8dec3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.203884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.203831 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.203884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.203859 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.203884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.203872 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.204105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.203896 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.204105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.203909 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.204728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.204702 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-token" pod "85cd456c-fb72-48b1-91e5-c46d41e8dec3" (UID: "85cd456c-fb72-48b1-91e5-c46d41e8dec3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:34.205148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.205126 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-envoy" pod "85cd456c-fb72-48b1-91e5-c46d41e8dec3" (UID: "85cd456c-fb72-48b1-91e5-c46d41e8dec3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.205340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.205309 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-kube-api-access-24f2v" pod "85cd456c-fb72-48b1-91e5-c46d41e8dec3" (UID: "85cd456c-fb72-48b1-91e5-c46d41e8dec3"). InnerVolumeSpecName "kube-api-access-24f2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:34.206142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.206120 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-podinfo" pod "85cd456c-fb72-48b1-91e5-c46d41e8dec3" (UID: "85cd456c-fb72-48b1-91e5-c46d41e8dec3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:34.305046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305015 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-data\") pod \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " Oct 29 23:19:34.305250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305064 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-socket\") pod \"882db266-6c50-4d21-a39d-1b6459694d44\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " Oct 29 23:19:34.305250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305105 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-socket\") pod \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " Oct 29 23:19:34.305250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305144 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-kube-api-access-nl2m7\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-kube-api-access-nl2m7\") pod \"882db266-6c50-4d21-a39d-1b6459694d44\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " Oct 29 23:19:34.305250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305184 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882db266-6c50-4d21-a39d-1b6459694d44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882db266-6c50-4d21-a39d-1b6459694d44-istiod-ca-cert\") pod \"882db266-6c50-4d21-a39d-1b6459694d44\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " Oct 29 23:19:34.305250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305245 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-envoy\") pod \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " Oct 29 23:19:34.305631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305288 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-credential-socket\") pod \"882db266-6c50-4d21-a39d-1b6459694d44\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " Oct 29 23:19:34.305631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305320 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-data" pod "ef13c0f4-a239-4b3b-9c83-c36db2353b19" (UID: "ef13c0f4-a239-4b3b-9c83-c36db2353b19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.305631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305366 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-credential-socket\") pod \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " Oct 29 23:19:34.305631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305383 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-socket" pod "ef13c0f4-a239-4b3b-9c83-c36db2353b19" (UID: "ef13c0f4-a239-4b3b-9c83-c36db2353b19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.305631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305409 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-socket" pod "882db266-6c50-4d21-a39d-1b6459694d44" (UID: "882db266-6c50-4d21-a39d-1b6459694d44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.305631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305407 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-certs\") pod \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " Oct 29 23:19:34.305631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305517 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-certs\") pod \"882db266-6c50-4d21-a39d-1b6459694d44\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " Oct 29 23:19:34.306021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305633 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-token\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-token\") pod \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " Oct 29 23:19:34.306021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305637 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-credential-socket" pod "ef13c0f4-a239-4b3b-9c83-c36db2353b19" (UID: "ef13c0f4-a239-4b3b-9c83-c36db2353b19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.306021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305674 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-data\") pod \"882db266-6c50-4d21-a39d-1b6459694d44\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " Oct 29 23:19:34.306021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305757 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-kube-api-access-4hj62\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-kube-api-access-4hj62\") pod \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " Oct 29 23:19:34.306021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305765 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-certs" pod "ef13c0f4-a239-4b3b-9c83-c36db2353b19" (UID: "ef13c0f4-a239-4b3b-9c83-c36db2353b19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.306021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305829 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-podinfo\") pod \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " Oct 29 23:19:34.306021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305872 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-istio-token\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-istio-token\") pod \"882db266-6c50-4d21-a39d-1b6459694d44\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " Oct 29 23:19:34.306021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.305950 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-envoy\") pod \"882db266-6c50-4d21-a39d-1b6459694d44\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " Oct 29 23:19:34.306431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.306046 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/882db266-6c50-4d21-a39d-1b6459694d44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882db266-6c50-4d21-a39d-1b6459694d44-istio-podinfo\") pod \"882db266-6c50-4d21-a39d-1b6459694d44\" (UID: \"882db266-6c50-4d21-a39d-1b6459694d44\") " Oct 29 23:19:34.306431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.306089 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istiod-ca-cert\") pod \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\" (UID: \"ef13c0f4-a239-4b3b-9c83-c36db2353b19\") " Oct 29 23:19:34.306431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.306180 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-credential-socket" pod "882db266-6c50-4d21-a39d-1b6459694d44" (UID: "882db266-6c50-4d21-a39d-1b6459694d44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.306431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.306190 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-certs" pod "882db266-6c50-4d21-a39d-1b6459694d44" (UID: "882db266-6c50-4d21-a39d-1b6459694d44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.306431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.306332 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882db266-6c50-4d21-a39d-1b6459694d44-istiod-ca-cert" pod "882db266-6c50-4d21-a39d-1b6459694d44" (UID: "882db266-6c50-4d21-a39d-1b6459694d44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:34.306727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.306665 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-data" pod "882db266-6c50-4d21-a39d-1b6459694d44" (UID: "882db266-6c50-4d21-a39d-1b6459694d44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.307045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.306998 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istiod-ca-cert" pod "ef13c0f4-a239-4b3b-9c83-c36db2353b19" (UID: "ef13c0f4-a239-4b3b-9c83-c36db2353b19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.309604 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24f2v\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-kube-api-access-24f2v\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.309637 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.309721 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.309752 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.309767 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.309780 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.309984 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882db266-6c50-4d21-a39d-1b6459694d44-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.310008 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.310022 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.310041 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.310074 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.310099 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.310099 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-podinfo" pod "ef13c0f4-a239-4b3b-9c83-c36db2353b19" (UID: "ef13c0f4-a239-4b3b-9c83-c36db2353b19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.310125 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cd456c-fb72-48b1-91e5-c46d41e8dec3-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.310141 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.326598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.325825 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-token" pod "ef13c0f4-a239-4b3b-9c83-c36db2353b19" (UID: "ef13c0f4-a239-4b3b-9c83-c36db2353b19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:34.333076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.333031 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/882db266-6c50-4d21-a39d-1b6459694d44-istio-podinfo" pod "882db266-6c50-4d21-a39d-1b6459694d44" (UID: "882db266-6c50-4d21-a39d-1b6459694d44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:34.333523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.333494 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-kube-api-access-4hj62" pod "ef13c0f4-a239-4b3b-9c83-c36db2353b19" (UID: "ef13c0f4-a239-4b3b-9c83-c36db2353b19"). InnerVolumeSpecName "kube-api-access-4hj62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:34.335246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.335223 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-envoy" pod "ef13c0f4-a239-4b3b-9c83-c36db2353b19" (UID: "ef13c0f4-a239-4b3b-9c83-c36db2353b19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.335330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.335288 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-kube-api-access-nl2m7" pod "882db266-6c50-4d21-a39d-1b6459694d44" (UID: "882db266-6c50-4d21-a39d-1b6459694d44"). InnerVolumeSpecName "kube-api-access-nl2m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:34.335454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.335423 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-envoy" pod "882db266-6c50-4d21-a39d-1b6459694d44" (UID: "882db266-6c50-4d21-a39d-1b6459694d44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:34.335557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.335486 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-istio-token" pod "882db266-6c50-4d21-a39d-1b6459694d44" (UID: "882db266-6c50-4d21-a39d-1b6459694d44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:34.411798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.411705 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2m7\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-kube-api-access-nl2m7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.411798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.411738 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.411798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.411754 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.411798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.411767 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hj62\" (UniqueName: \"kubernetes.io/projected/ef13c0f4-a239-4b3b-9c83-c36db2353b19-kube-api-access-4hj62\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.412153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.411782 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef13c0f4-a239-4b3b-9c83-c36db2353b19-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.412153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.411845 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882db266-6c50-4d21-a39d-1b6459694d44-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.412153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.411860 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882db266-6c50-4d21-a39d-1b6459694d44-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.412153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.411891 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882db266-6c50-4d21-a39d-1b6459694d44-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:34.445458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.445425 5736 generic.go:356] "Generic (PLEG): container finished" podID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerID="5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e" exitCode=0 Oct 29 23:19:34.445670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.445500 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" Oct 29 23:19:34.445670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.445511 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" event={"ID":"85cd456c-fb72-48b1-91e5-c46d41e8dec3","Type":"ContainerDied","Data":"5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e"} Oct 29 23:19:34.445670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.445621 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2" event={"ID":"85cd456c-fb72-48b1-91e5-c46d41e8dec3","Type":"ContainerDied","Data":"c7bb426e850fa5bcf7b6b77c94d220208bb701e6e1b9e21def478ea07f6e48db"} Oct 29 23:19:34.445670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.445651 5736 scope.go:122] "RemoveContainer" containerID="5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e" Oct 29 23:19:34.447262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.447228 5736 generic.go:356] "Generic (PLEG): container finished" podID="882db266-6c50-4d21-a39d-1b6459694d44" containerID="49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139" exitCode=0 Oct 29 23:19:34.447375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.447320 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" Oct 29 23:19:34.447456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.447440 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" event={"ID":"882db266-6c50-4d21-a39d-1b6459694d44","Type":"ContainerDied","Data":"49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139"} Oct 29 23:19:34.447506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.447481 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw" event={"ID":"882db266-6c50-4d21-a39d-1b6459694d44","Type":"ContainerDied","Data":"acf4cd00033e956fd7d8bb6921865f84ee9fe8cf48cbc2c2d01ad10203af8bcd"} Oct 29 23:19:34.449284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.449261 5736 generic.go:356] "Generic (PLEG): container finished" podID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerID="7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a" exitCode=0 Oct 29 23:19:34.449411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.449298 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" event={"ID":"ef13c0f4-a239-4b3b-9c83-c36db2353b19","Type":"ContainerDied","Data":"7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a"} Oct 29 23:19:34.449411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.449326 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" Oct 29 23:19:34.449411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.449350 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r" event={"ID":"ef13c0f4-a239-4b3b-9c83-c36db2353b19","Type":"ContainerDied","Data":"11e2b0bbb60b7ebbb16b093048da671ca3f61e27d31207c074f249f728f39f55"} Oct 29 23:19:34.458934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.458915 5736 scope.go:122] "RemoveContainer" containerID="5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e" Oct 29 23:19:34.459289 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:19:34.459268 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e\": container with ID starting with 5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e not found: ID does not exist" containerID="5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e" Oct 29 23:19:34.459362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.459297 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e"} err="failed to get container status \"5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e\": rpc error: code = NotFound desc = could not find container \"5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e\": container with ID starting with 5ceacbe6d78d8b7bf461a20ccdbe5b10690c8dc52d1758deb72d5db751ee054e not found: ID does not exist" Oct 29 23:19:34.459362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.459314 5736 scope.go:122] "RemoveContainer" containerID="49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139" Oct 29 23:19:34.469157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.469135 5736 scope.go:122] "RemoveContainer" containerID="49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139" Oct 29 23:19:34.469434 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:19:34.469414 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139\": container with ID starting with 49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139 not found: ID does not exist" containerID="49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139" Oct 29 23:19:34.469504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.469440 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139"} err="failed to get container status \"49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139\": rpc error: code = NotFound desc = could not find container \"49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139\": container with ID starting with 49057429a65f96f07391ce8762dc86071cf0079781ba5e4669646bc8c015f139 not found: ID does not exist" Oct 29 23:19:34.469504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.469455 5736 scope.go:122] "RemoveContainer" containerID="7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a" Oct 29 23:19:34.479199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.479177 5736 scope.go:122] "RemoveContainer" containerID="7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a" Oct 29 23:19:34.479457 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:19:34.479434 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a\": container with ID starting with 7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a not found: ID does not exist" containerID="7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a" Oct 29 23:19:34.479530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.479473 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a"} err="failed to get container status \"7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a\": rpc error: code = NotFound desc = could not find container \"7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a\": container with ID starting with 7b707639ba5a4b03e0eebd399a276c07e0ebdd91f14758b6f7bbc79b683c662a not found: ID does not exist" Oct 29 23:19:34.523179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.523154 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2"] Oct 29 23:19:34.535313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.535291 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-p9dp2"] Oct 29 23:19:34.570356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.570334 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r"] Oct 29 23:19:34.574483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.574461 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-pl72r"] Oct 29 23:19:34.603443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.603422 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw"] Oct 29 23:19:34.607647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:34.607619 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-llcxw"] Oct 29 23:19:35.004311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:35.004282 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" path="/var/lib/kubelet/pods/85cd456c-fb72-48b1-91e5-c46d41e8dec3/volumes" Oct 29 23:19:35.004797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:35.004782 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882db266-6c50-4d21-a39d-1b6459694d44" path="/var/lib/kubelet/pods/882db266-6c50-4d21-a39d-1b6459694d44/volumes" Oct 29 23:19:35.005246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:35.005232 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" path="/var/lib/kubelet/pods/ef13c0f4-a239-4b3b-9c83-c36db2353b19/volumes" Oct 29 23:19:39.887270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:39.887225 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" podUID="6234f8b4-47e6-47e3-9f0b-dba6cccc5434" containerName="main" containerID="cri-o://3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0" gracePeriod=45 Oct 29 23:19:40.181056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.181030 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:19:40.301666 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.301633 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-model-cache\") pod \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " Oct 29 23:19:40.301833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.301749 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tmp-dir\") pod \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " Oct 29 23:19:40.301833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.301794 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-kube-api-access-plfh7\" (UniqueName: \"kubernetes.io/projected/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-kube-api-access-plfh7\") pod \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " Oct 29 23:19:40.301833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.301815 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tls-certs\" (UniqueName: \"kubernetes.io/secret/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tls-certs\") pod \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " Oct 29 23:19:40.301952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.301846 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-dshm\") pod \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " Oct 29 23:19:40.301952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.301899 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-model-cache" pod "6234f8b4-47e6-47e3-9f0b-dba6cccc5434" (UID: "6234f8b4-47e6-47e3-9f0b-dba6cccc5434"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.302121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.301949 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-home\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-home\") pod \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\" (UID: \"6234f8b4-47e6-47e3-9f0b-dba6cccc5434\") " Oct 29 23:19:40.302121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.302046 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tmp-dir" pod "6234f8b4-47e6-47e3-9f0b-dba6cccc5434" (UID: "6234f8b4-47e6-47e3-9f0b-dba6cccc5434"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.302244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.302233 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-home" pod "6234f8b4-47e6-47e3-9f0b-dba6cccc5434" (UID: "6234f8b4-47e6-47e3-9f0b-dba6cccc5434"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.302643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.302624 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.302643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.302642 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.302917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.302652 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.304012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.303986 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-dshm" pod "6234f8b4-47e6-47e3-9f0b-dba6cccc5434" (UID: "6234f8b4-47e6-47e3-9f0b-dba6cccc5434"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:40.304117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.304094 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tls-certs" pod "6234f8b4-47e6-47e3-9f0b-dba6cccc5434" (UID: "6234f8b4-47e6-47e3-9f0b-dba6cccc5434"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:19:40.304162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.304146 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-kube-api-access-plfh7" pod "6234f8b4-47e6-47e3-9f0b-dba6cccc5434" (UID: "6234f8b4-47e6-47e3-9f0b-dba6cccc5434"). InnerVolumeSpecName "kube-api-access-plfh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:40.407116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.407036 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plfh7\" (UniqueName: \"kubernetes.io/projected/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-kube-api-access-plfh7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.407300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.407286 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.407404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.407384 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6234f8b4-47e6-47e3-9f0b-dba6cccc5434-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:19:40.496495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.496461 5736 generic.go:356] "Generic (PLEG): container finished" podID="6234f8b4-47e6-47e3-9f0b-dba6cccc5434" containerID="3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0" exitCode=0 Oct 29 23:19:40.496680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.496504 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" Oct 29 23:19:40.496680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.496535 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" event={"ID":"6234f8b4-47e6-47e3-9f0b-dba6cccc5434","Type":"ContainerDied","Data":"3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0"} Oct 29 23:19:40.496680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.496611 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk" event={"ID":"6234f8b4-47e6-47e3-9f0b-dba6cccc5434","Type":"ContainerDied","Data":"f9e7c16e28baf7fe4127dbb453a2638af82fc67effc6b90209a344d15ece2f12"} Oct 29 23:19:40.496680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.496632 5736 scope.go:122] "RemoveContainer" containerID="3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0" Oct 29 23:19:40.507945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.507907 5736 scope.go:122] "RemoveContainer" containerID="3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0" Oct 29 23:19:40.508248 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:19:40.508216 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0\": container with ID starting with 3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0 not found: ID does not exist" containerID="3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0" Oct 29 23:19:40.508352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.508255 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0"} err="failed to get container status \"3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0\": rpc error: code = NotFound desc = could not find container \"3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0\": container with ID starting with 3d6cedbcfeb42809d69ad76753cde71bcb9734954bc02bbe1a4e38b8e4a205c0 not found: ID does not exist" Oct 29 23:19:40.542031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.542010 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk"] Oct 29 23:19:40.546026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:40.546008 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-96cd49/praxis-route-96cd49-kserve-fb645f9d7-vzzwk"] Oct 29 23:19:41.005163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:19:41.005127 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6234f8b4-47e6-47e3-9f0b-dba6cccc5434" path="/var/lib/kubelet/pods/6234f8b4-47e6-47e3-9f0b-dba6cccc5434/volumes" Oct 29 23:19:56.824728 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:19:56.824689 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:20:10.387970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.387928 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb"] Oct 29 23:20:10.388571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.388165 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" containerID="cri-o://c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab" gracePeriod=30 Oct 29 23:20:10.967175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967141 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276"] Oct 29 23:20:10.967774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967758 5736 cpu_manager.go:414] "containerMap: removing container" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" Oct 29 23:20:10.967774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967774 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967797 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5f82ec59-ff49-45ed-ad06-c7c88ced15f0" containerName="revoke-keys" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967802 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f82ec59-ff49-45ed-ad06-c7c88ced15f0" containerName="revoke-keys" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967822 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967828 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967838 5736 cpu_manager.go:414] "containerMap: removing container" podUID="138b5b65-38aa-4c67-916f-1047e5898300" containerName="istio-proxy" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967842 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138b5b65-38aa-4c67-916f-1047e5898300" containerName="istio-proxy" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967854 5736 cpu_manager.go:414] "containerMap: removing container" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967859 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967875 5736 cpu_manager.go:414] "containerMap: removing container" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967882 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" Oct 29 23:20:10.967888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967893 5736 cpu_manager.go:414] "containerMap: removing container" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967898 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967913 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6234f8b4-47e6-47e3-9f0b-dba6cccc5434" containerName="main" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967917 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6234f8b4-47e6-47e3-9f0b-dba6cccc5434" containerName="main" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967926 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967931 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967938 5736 cpu_manager.go:414] "containerMap: removing container" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.967942 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968026 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="de92d227-c2af-4a2c-92bc-6c85b4e77056" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968034 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bbb4ac-5639-4e05-8260-6ef27d0e526c" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968044 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6234f8b4-47e6-47e3-9f0b-dba6cccc5434" containerName="main" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968049 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abd5efc-ca12-4692-bc81-e62dadfd60a7" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968056 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="882db266-6c50-4d21-a39d-1b6459694d44" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968062 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="85cd456c-fb72-48b1-91e5-c46d41e8dec3" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968069 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f82ec59-ff49-45ed-ad06-c7c88ced15f0" containerName="revoke-keys" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968077 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef13c0f4-a239-4b3b-9c83-c36db2353b19" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968084 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="138b5b65-38aa-4c67-916f-1047e5898300" containerName="istio-proxy" Oct 29 23:20:10.968234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.968089 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d85aebb-8ada-4414-8663-4af63145e5ce" containerName="istio-proxy" Oct 29 23:20:10.971704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.971679 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:10.987856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.987830 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:20:10.988075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.988059 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:20:10.992932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:10.992904 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276"] Oct 29 23:20:11.109853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.109822 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.110030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.109897 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.110030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.109969 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22htg\" (UniqueName: \"kubernetes.io/projected/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-kube-api-access-22htg\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.210968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.210931 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.211159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.211013 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.211159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.211043 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22htg\" (UniqueName: \"kubernetes.io/projected/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-kube-api-access-22htg\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.211773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.211750 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.213376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.213359 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.231978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.231925 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22htg\" (UniqueName: \"kubernetes.io/projected/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-kube-api-access-22htg\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.291780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.291740 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:11.616883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.616597 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276"] Oct 29 23:20:11.620036 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:20:11.620009 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628edfb4_3c57_42db_bcdb_e0cefb63ac8b.slice/crio-c8ede8dc30303ffe8f159dab9873ee13e503ae5b527ecaefb32e2c00a27b4746 WatchSource:0}: Error finding container c8ede8dc30303ffe8f159dab9873ee13e503ae5b527ecaefb32e2c00a27b4746: Status 404 returned error can't find the container with id c8ede8dc30303ffe8f159dab9873ee13e503ae5b527ecaefb32e2c00a27b4746 Oct 29 23:20:11.726850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:11.726821 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" event={"ID":"628edfb4-3c57-42db-bcdb-e0cefb63ac8b","Type":"ContainerStarted","Data":"c8ede8dc30303ffe8f159dab9873ee13e503ae5b527ecaefb32e2c00a27b4746"} Oct 29 23:20:12.735536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:12.735501 5736 generic.go:356] "Generic (PLEG): container finished" podID="628edfb4-3c57-42db-bcdb-e0cefb63ac8b" containerID="0f490b320ba761f33e205ae0899134e17bc93ffc0cb8b708b248283fe5da59f4" exitCode=0 Oct 29 23:20:12.735942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:12.735618 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" event={"ID":"628edfb4-3c57-42db-bcdb-e0cefb63ac8b","Type":"ContainerDied","Data":"0f490b320ba761f33e205ae0899134e17bc93ffc0cb8b708b248283fe5da59f4"} Oct 29 23:20:13.914403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:13.914377 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:14.052550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.052478 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-kube-api-access-22htg\" (UniqueName: \"kubernetes.io/projected/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-kube-api-access-22htg\") pod \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " Oct 29 23:20:14.052550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.052546 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-openshift-ca\") pod \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " Oct 29 23:20:14.052744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.052630 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-xks-ca\") pod \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\" (UID: \"628edfb4-3c57-42db-bcdb-e0cefb63ac8b\") " Oct 29 23:20:14.052938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.052911 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-openshift-ca" pod "628edfb4-3c57-42db-bcdb-e0cefb63ac8b" (UID: "628edfb4-3c57-42db-bcdb-e0cefb63ac8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:20:14.053267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.053247 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:14.054724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.054702 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-xks-ca" pod "628edfb4-3c57-42db-bcdb-e0cefb63ac8b" (UID: "628edfb4-3c57-42db-bcdb-e0cefb63ac8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:20:14.054829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.054718 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-kube-api-access-22htg" pod "628edfb4-3c57-42db-bcdb-e0cefb63ac8b" (UID: "628edfb4-3c57-42db-bcdb-e0cefb63ac8b"). InnerVolumeSpecName "kube-api-access-22htg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:14.154257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.154227 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22htg\" (UniqueName: \"kubernetes.io/projected/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-kube-api-access-22htg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:14.154257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.154254 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628edfb4-3c57-42db-bcdb-e0cefb63ac8b-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:14.750710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.750673 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" Oct 29 23:20:14.750710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.750682 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276" event={"ID":"628edfb4-3c57-42db-bcdb-e0cefb63ac8b","Type":"ContainerDied","Data":"c8ede8dc30303ffe8f159dab9873ee13e503ae5b527ecaefb32e2c00a27b4746"} Oct 29 23:20:14.750951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:14.750734 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8ede8dc30303ffe8f159dab9873ee13e503ae5b527ecaefb32e2c00a27b4746" Oct 29 23:20:15.703302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.703276 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:20:15.762724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.762686 5736 generic.go:356] "Generic (PLEG): container finished" podID="dde8163b-c690-4511-ad4d-ba7acee99494" containerID="c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab" exitCode=0 Oct 29 23:20:15.762908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.762732 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" event={"ID":"dde8163b-c690-4511-ad4d-ba7acee99494","Type":"ContainerDied","Data":"c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab"} Oct 29 23:20:15.762908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.762735 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" Oct 29 23:20:15.762908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.762818 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb" event={"ID":"dde8163b-c690-4511-ad4d-ba7acee99494","Type":"ContainerDied","Data":"a577ef9614d5446112bf094b9aae6aa1b6e14ee30bc2e0ffb70540751bb55c63"} Oct 29 23:20:15.762908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.762848 5736 scope.go:122] "RemoveContainer" containerID="c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab" Oct 29 23:20:15.778559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.778534 5736 scope.go:122] "RemoveContainer" containerID="c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab" Oct 29 23:20:15.778958 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:20:15.778925 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab\": container with ID starting with c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab not found: ID does not exist" containerID="c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab" Oct 29 23:20:15.779066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.778971 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab"} err="failed to get container status \"c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab\": rpc error: code = NotFound desc = could not find container \"c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab\": container with ID starting with c0560b030aa4977ef244f4a620c51b82b4ab80db6edbb167083e59873a1199ab not found: ID does not exist" Oct 29 23:20:15.879196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879165 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde8163b-c690-4511-ad4d-ba7acee99494-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde8163b-c690-4511-ad4d-ba7acee99494-istiod-ca-cert\") pod \"dde8163b-c690-4511-ad4d-ba7acee99494\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " Oct 29 23:20:15.879363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879243 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-socket\") pod \"dde8163b-c690-4511-ad4d-ba7acee99494\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " Oct 29 23:20:15.879363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879279 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-data\") pod \"dde8163b-c690-4511-ad4d-ba7acee99494\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " Oct 29 23:20:15.879363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879306 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-certs\") pod \"dde8163b-c690-4511-ad4d-ba7acee99494\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " Oct 29 23:20:15.879363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879331 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-istio-token\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-istio-token\") pod \"dde8163b-c690-4511-ad4d-ba7acee99494\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " Oct 29 23:20:15.879625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879417 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dde8163b-c690-4511-ad4d-ba7acee99494-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde8163b-c690-4511-ad4d-ba7acee99494-istio-podinfo\") pod \"dde8163b-c690-4511-ad4d-ba7acee99494\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " Oct 29 23:20:15.879625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879505 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-envoy\") pod \"dde8163b-c690-4511-ad4d-ba7acee99494\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " Oct 29 23:20:15.879625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879608 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-credential-socket\") pod \"dde8163b-c690-4511-ad4d-ba7acee99494\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " Oct 29 23:20:15.879800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879614 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-data" pod "dde8163b-c690-4511-ad4d-ba7acee99494" (UID: "dde8163b-c690-4511-ad4d-ba7acee99494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.879800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879632 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde8163b-c690-4511-ad4d-ba7acee99494-istiod-ca-cert" pod "dde8163b-c690-4511-ad4d-ba7acee99494" (UID: "dde8163b-c690-4511-ad4d-ba7acee99494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:20:15.879800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879690 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-kube-api-access-6g64j\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-kube-api-access-6g64j\") pod \"dde8163b-c690-4511-ad4d-ba7acee99494\" (UID: \"dde8163b-c690-4511-ad4d-ba7acee99494\") " Oct 29 23:20:15.879800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879745 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-certs" pod "dde8163b-c690-4511-ad4d-ba7acee99494" (UID: "dde8163b-c690-4511-ad4d-ba7acee99494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.880025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879954 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-credential-socket" pod "dde8163b-c690-4511-ad4d-ba7acee99494" (UID: "dde8163b-c690-4511-ad4d-ba7acee99494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.880025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.879984 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-socket" pod "dde8163b-c690-4511-ad4d-ba7acee99494" (UID: "dde8163b-c690-4511-ad4d-ba7acee99494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.880412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.880387 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde8163b-c690-4511-ad4d-ba7acee99494-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.880536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.880416 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.880536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.880433 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.880536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.880448 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.880536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.880461 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.881920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.881898 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-kube-api-access-6g64j" pod "dde8163b-c690-4511-ad4d-ba7acee99494" (UID: "dde8163b-c690-4511-ad4d-ba7acee99494"). InnerVolumeSpecName "kube-api-access-6g64j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:15.882035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.882018 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dde8163b-c690-4511-ad4d-ba7acee99494-istio-podinfo" pod "dde8163b-c690-4511-ad4d-ba7acee99494" (UID: "dde8163b-c690-4511-ad4d-ba7acee99494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:20:15.882143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.882121 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-envoy" pod "dde8163b-c690-4511-ad4d-ba7acee99494" (UID: "dde8163b-c690-4511-ad4d-ba7acee99494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:15.882244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.882192 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-istio-token" pod "dde8163b-c690-4511-ad4d-ba7acee99494" (UID: "dde8163b-c690-4511-ad4d-ba7acee99494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:15.981937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.981906 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde8163b-c690-4511-ad4d-ba7acee99494-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.981937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.981932 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde8163b-c690-4511-ad4d-ba7acee99494-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.981937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.981943 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g64j\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-kube-api-access-6g64j\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:15.982162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:15.981952 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde8163b-c690-4511-ad4d-ba7acee99494-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:20:16.106470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:16.106438 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb"] Oct 29 23:20:16.116253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:16.116228 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-96cd49-openshift-default-7c8ddf6b69-6b7tb"] Oct 29 23:20:17.007456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:17.007424 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" path="/var/lib/kubelet/pods/dde8163b-c690-4511-ad4d-ba7acee99494/volumes" Oct 29 23:20:26.828359 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:20:26.828323 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:20:39.470091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.470059 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f"] Oct 29 23:20:39.470768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.470750 5736 cpu_manager.go:414] "containerMap: removing container" podUID="628edfb4-3c57-42db-bcdb-e0cefb63ac8b" containerName="revoke-keys" Oct 29 23:20:39.470768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.470768 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628edfb4-3c57-42db-bcdb-e0cefb63ac8b" containerName="revoke-keys" Oct 29 23:20:39.470910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.470800 5736 cpu_manager.go:414] "containerMap: removing container" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" Oct 29 23:20:39.470910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.470807 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" Oct 29 23:20:39.470991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.470915 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde8163b-c690-4511-ad4d-ba7acee99494" containerName="istio-proxy" Oct 29 23:20:39.470991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.470932 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="628edfb4-3c57-42db-bcdb-e0cefb63ac8b" containerName="revoke-keys" Oct 29 23:20:39.476309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.476276 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.491857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.490904 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26b322\"/\"routing-overlay\"" Oct 29 23:20:39.496613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.496566 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f"] Oct 29 23:20:39.500832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.500808 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26b322\"/\"kube-root-ca.crt\"" Oct 29 23:20:39.501004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.500982 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26b322\"/\"payload-processing-external-model-plugins-e2e-extmod-b-26b322\"" Oct 29 23:20:39.501187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.501167 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26b322\"/\"openshift-service-ca.crt\"" Oct 29 23:20:39.501311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.501211 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26b322\"/\"agc-cred-6b2c5c36e32c\"" Oct 29 23:20:39.501311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.501262 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26b322\"/\"payload-processing-external-model-e2e--bf3bb98d-dockercfg-mvn6w\"" Oct 29 23:20:39.559987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.559944 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.560178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.560087 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.560178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.560150 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.560328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.560312 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnww\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-kube-api-access-ddnww\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.661296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.661263 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnww\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-kube-api-access-ddnww\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.661518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.661328 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.661518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.661402 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.661518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.661425 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.662182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.662147 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.662302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.662246 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.664166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.664141 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.683468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.683433 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnww\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-kube-api-access-ddnww\") pod \"payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:39.805373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:39.805287 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:40.123237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:40.123210 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f"] Oct 29 23:20:40.125973 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:20:40.125947 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc7de1d_080d_4700_9a10_88583217467f.slice/crio-10daac6c5b45bf730d5b1f5d0d049babbcb797b7fb166d55b1d1165bd825d3b8 WatchSource:0}: Error finding container 10daac6c5b45bf730d5b1f5d0d049babbcb797b7fb166d55b1d1165bd825d3b8: Status 404 returned error can't find the container with id 10daac6c5b45bf730d5b1f5d0d049babbcb797b7fb166d55b1d1165bd825d3b8 Oct 29 23:20:40.967079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:40.967042 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" event={"ID":"cdc7de1d-080d-4700-9a10-88583217467f","Type":"ContainerStarted","Data":"870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0"} Oct 29 23:20:40.967535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:40.967106 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" event={"ID":"cdc7de1d-080d-4700-9a10-88583217467f","Type":"ContainerStarted","Data":"10daac6c5b45bf730d5b1f5d0d049babbcb797b7fb166d55b1d1165bd825d3b8"} Oct 29 23:20:40.967535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:40.967146 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:41.006012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:41.005954 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" podStartSLOduration=2.005935533 podStartE2EDuration="2.005935533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:41.004220564 +0000 UTC m=+2238.120140170" watchObservedRunningTime="2026-10-29 23:20:41.005935533 +0000 UTC m=+2238.121855140" Oct 29 23:20:51.976991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:51.976961 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:20:56.824208 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:20:56.824168 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:20:57.753134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.753099 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb"] Oct 29 23:20:57.757483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.757462 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:57.773693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.773671 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7a3c1a-openshift-default-dockercfg-69xwl\"" Oct 29 23:20:57.788311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.788280 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb"] Oct 29 23:20:57.916965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.916928 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:57.917371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.916982 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:57.917371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.917112 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:57.917371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.917185 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:57.917371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.917220 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:57.917513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.917378 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:57.917513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.917431 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f43a283-0121-4466-8da1-eb3c861a59dd-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:57.917513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.917497 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kc9d\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-kube-api-access-7kc9d\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:57.917631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:57.917527 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019065 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019110 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019228 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019278 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f43a283-0121-4466-8da1-eb3c861a59dd-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019311 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kc9d\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-kube-api-access-7kc9d\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019340 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019423 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019466 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019519 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019543 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019558 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.019948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.019772 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.020040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.020018 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f43a283-0121-4466-8da1-eb3c861a59dd-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.020091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.020026 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.021516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.021490 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.022105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.022081 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.040538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.040518 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.040836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.040818 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kc9d\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-kube-api-access-7kc9d\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.073959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.073938 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:20:58.396887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.396798 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb"] Oct 29 23:20:58.409743 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:20:58.409713 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f43a283_0121_4466_8da1_eb3c861a59dd.slice/crio-e49f9a7c3913f717fdfd742b6e81b002022d523d4c59e198cac657b54bf77eb3 WatchSource:0}: Error finding container e49f9a7c3913f717fdfd742b6e81b002022d523d4c59e198cac657b54bf77eb3: Status 404 returned error can't find the container with id e49f9a7c3913f717fdfd742b6e81b002022d523d4c59e198cac657b54bf77eb3 Oct 29 23:20:58.413684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.413635 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:20:58.413802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.413718 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:20:58.413802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:58.413757 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:20:59.116727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:59.116684 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" event={"ID":"3f43a283-0121-4466-8da1-eb3c861a59dd","Type":"ContainerStarted","Data":"a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400"} Oct 29 23:20:59.117237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:59.116740 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" event={"ID":"3f43a283-0121-4466-8da1-eb3c861a59dd","Type":"ContainerStarted","Data":"e49f9a7c3913f717fdfd742b6e81b002022d523d4c59e198cac657b54bf77eb3"} Oct 29 23:20:59.167441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:20:59.167392 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" podStartSLOduration=2.167378541 podStartE2EDuration="2.167378541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:59.167179924 +0000 UTC m=+2256.283099529" watchObservedRunningTime="2026-10-29 23:20:59.167378541 +0000 UTC m=+2256.283298208" Oct 29 23:21:00.074588 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:00.074544 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:21:00.076004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:00.075975 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:21:00.076140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:00.076035 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 23:21:01.074379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:01.074338 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:21:01.074865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:01.074420 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 23:21:02.075203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:02.075165 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:21:02.075628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:02.075239 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 23:21:03.189757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:03.189720 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:21:03.190334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:03.190302 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 23:21:03.619642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:03.619597 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276"] Oct 29 23:21:03.628677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:03.628622 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a3c1a-zt276"] Oct 29 23:21:04.075237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:04.075194 5736 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 29 23:21:04.075458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:04.075259 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 29 23:21:05.005667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:05.005632 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628edfb4-3c57-42db-bcdb-e0cefb63ac8b" path="/var/lib/kubelet/pods/628edfb4-3c57-42db-bcdb-e0cefb63ac8b/volumes" Oct 29 23:21:05.078934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:05.078901 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:21:05.079373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:05.079357 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:21:05.080057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:05.080037 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:21:08.361293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.361255 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk"] Oct 29 23:21:08.366724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.366698 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.381465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.380917 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-206095-openshift-default-dockercfg-69qxn\"" Oct 29 23:21:08.420609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.395433 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk"] Oct 29 23:21:08.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.424522 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istiod-ca-cert\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.424594 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-envoy\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.424630 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-certs\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.424733 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-data\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.424775 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-socket\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.424812 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqxxf\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-kube-api-access-xqxxf\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.425033 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-credential-socket\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.425078 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-token\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.425598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.425131 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-podinfo\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527179 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-credential-socket\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527230 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-token\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527265 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-podinfo\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527284 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istiod-ca-cert\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527312 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-envoy\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527343 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-certs\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527453 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-data\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527491 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-socket\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527530 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqxxf\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-kube-api-access-xqxxf\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.527768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527646 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-credential-socket\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.528008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.527937 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-certs\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.528072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.528014 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istiod-ca-cert\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.528072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.528056 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-socket\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.528230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.528198 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-data\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.529915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.529894 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-podinfo\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.530135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.530113 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-envoy\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.549030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.548989 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqxxf\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-kube-api-access-xqxxf\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.549352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.549327 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-token\") pod \"e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:08.716632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:08.705884 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:09.111042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:09.111012 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk"] Oct 29 23:21:09.114828 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:09.114779 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4e89eb_8c53_4fcd_ab11_2fe91e3d3666.slice/crio-4cb59518417031ec465da86fa5eb489a062ba2ef94194a273c9d05db8bd2b7f1 WatchSource:0}: Error finding container 4cb59518417031ec465da86fa5eb489a062ba2ef94194a273c9d05db8bd2b7f1: Status 404 returned error can't find the container with id 4cb59518417031ec465da86fa5eb489a062ba2ef94194a273c9d05db8bd2b7f1 Oct 29 23:21:09.118001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:09.117964 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:21:09.118148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:09.118032 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:21:09.118148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:09.118060 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:21:09.259453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:09.259411 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" event={"ID":"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666","Type":"ContainerStarted","Data":"faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680"} Oct 29 23:21:09.259702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:09.259482 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" event={"ID":"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666","Type":"ContainerStarted","Data":"4cb59518417031ec465da86fa5eb489a062ba2ef94194a273c9d05db8bd2b7f1"} Oct 29 23:21:09.301883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:09.301789 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" podStartSLOduration=1.301769588 podStartE2EDuration="1.301769588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:09.300865507 +0000 UTC m=+2266.416785112" watchObservedRunningTime="2026-10-29 23:21:09.301769588 +0000 UTC m=+2266.417689196" Oct 29 23:21:09.706454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:09.706414 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:10.681878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:10.681842 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb"] Oct 29 23:21:10.682485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:10.682413 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerName="istio-proxy" containerID="cri-o://a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400" gracePeriod=30 Oct 29 23:21:10.714251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:10.714221 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:11.281298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:11.281269 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:11.282439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:11.282418 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:21:13.486726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.486692 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb"] Oct 29 23:21:13.491410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.491388 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:13.514283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.514246 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-206095\"" Oct 29 23:21:13.514438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.514280 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-206095-dockercfg-h9g5v\"" Oct 29 23:21:13.518253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.518229 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb"] Oct 29 23:21:13.534709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.534682 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6p8\" (UniqueName: \"kubernetes.io/projected/36aa37e0-2bd6-411e-9483-e79875d4efba-kube-api-access-hl6p8\") pod \"payload-processing-e2e-extmod-b-206095-7775f4646-tthgb\" (UID: \"36aa37e0-2bd6-411e-9483-e79875d4efba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:13.534866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.534842 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36aa37e0-2bd6-411e-9483-e79875d4efba-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-206095-7775f4646-tthgb\" (UID: \"36aa37e0-2bd6-411e-9483-e79875d4efba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:13.673602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.662252 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36aa37e0-2bd6-411e-9483-e79875d4efba-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-206095-7775f4646-tthgb\" (UID: \"36aa37e0-2bd6-411e-9483-e79875d4efba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:13.708603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.691026 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6p8\" (UniqueName: \"kubernetes.io/projected/36aa37e0-2bd6-411e-9483-e79875d4efba-kube-api-access-hl6p8\") pod \"payload-processing-e2e-extmod-b-206095-7775f4646-tthgb\" (UID: \"36aa37e0-2bd6-411e-9483-e79875d4efba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:13.709065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.665838 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36aa37e0-2bd6-411e-9483-e79875d4efba-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-206095-7775f4646-tthgb\" (UID: \"36aa37e0-2bd6-411e-9483-e79875d4efba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:13.779367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.779272 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6p8\" (UniqueName: \"kubernetes.io/projected/36aa37e0-2bd6-411e-9483-e79875d4efba-kube-api-access-hl6p8\") pod \"payload-processing-e2e-extmod-b-206095-7775f4646-tthgb\" (UID: \"36aa37e0-2bd6-411e-9483-e79875d4efba\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:13.807691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:13.807658 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:14.117484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:14.117455 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb"] Oct 29 23:21:14.120593 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:14.120551 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36aa37e0_2bd6_411e_9483_e79875d4efba.slice/crio-3327ae4ac6e1076e84ca5a9a874fe0b23dc07b3ec2dafff51747da6862b37244 WatchSource:0}: Error finding container 3327ae4ac6e1076e84ca5a9a874fe0b23dc07b3ec2dafff51747da6862b37244: Status 404 returned error can't find the container with id 3327ae4ac6e1076e84ca5a9a874fe0b23dc07b3ec2dafff51747da6862b37244 Oct 29 23:21:14.318278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:14.318225 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" event={"ID":"36aa37e0-2bd6-411e-9483-e79875d4efba","Type":"ContainerStarted","Data":"d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690"} Oct 29 23:21:14.318278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:14.318285 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" event={"ID":"36aa37e0-2bd6-411e-9483-e79875d4efba","Type":"ContainerStarted","Data":"3327ae4ac6e1076e84ca5a9a874fe0b23dc07b3ec2dafff51747da6862b37244"} Oct 29 23:21:14.318539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:14.318361 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:14.355757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:14.355709 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" podStartSLOduration=1.355691247 podStartE2EDuration="1.355691247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:14.353644226 +0000 UTC m=+2271.469563829" watchObservedRunningTime="2026-10-29 23:21:14.355691247 +0000 UTC m=+2271.471610854" Oct 29 23:21:15.983820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:15.983791 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:21:16.144864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.144824 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-podinfo\") pod \"3f43a283-0121-4466-8da1-eb3c861a59dd\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " Oct 29 23:21:16.145059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.144939 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-kube-api-access-7kc9d\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-kube-api-access-7kc9d\") pod \"3f43a283-0121-4466-8da1-eb3c861a59dd\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " Oct 29 23:21:16.145059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.145016 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-credential-socket\") pod \"3f43a283-0121-4466-8da1-eb3c861a59dd\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " Oct 29 23:21:16.145439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.145130 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f43a283-0121-4466-8da1-eb3c861a59dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f43a283-0121-4466-8da1-eb3c861a59dd-istiod-ca-cert\") pod \"3f43a283-0121-4466-8da1-eb3c861a59dd\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " Oct 29 23:21:16.145439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.145183 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-token\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-token\") pod \"3f43a283-0121-4466-8da1-eb3c861a59dd\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " Oct 29 23:21:16.145439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.145237 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-socket\") pod \"3f43a283-0121-4466-8da1-eb3c861a59dd\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " Oct 29 23:21:16.145439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.145352 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-certs\") pod \"3f43a283-0121-4466-8da1-eb3c861a59dd\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " Oct 29 23:21:16.145439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.145425 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-data\") pod \"3f43a283-0121-4466-8da1-eb3c861a59dd\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " Oct 29 23:21:16.145774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.145562 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-envoy\") pod \"3f43a283-0121-4466-8da1-eb3c861a59dd\" (UID: \"3f43a283-0121-4466-8da1-eb3c861a59dd\") " Oct 29 23:21:16.145930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.145896 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-certs" pod "3f43a283-0121-4466-8da1-eb3c861a59dd" (UID: "3f43a283-0121-4466-8da1-eb3c861a59dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.146132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.146091 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-socket" pod "3f43a283-0121-4466-8da1-eb3c861a59dd" (UID: "3f43a283-0121-4466-8da1-eb3c861a59dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.146451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.146419 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.146558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.146459 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.146717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.146600 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-data" pod "3f43a283-0121-4466-8da1-eb3c861a59dd" (UID: "3f43a283-0121-4466-8da1-eb3c861a59dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.146964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.146945 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-credential-socket" pod "3f43a283-0121-4466-8da1-eb3c861a59dd" (UID: "3f43a283-0121-4466-8da1-eb3c861a59dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.147081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.147066 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f43a283-0121-4466-8da1-eb3c861a59dd-istiod-ca-cert" pod "3f43a283-0121-4466-8da1-eb3c861a59dd" (UID: "3f43a283-0121-4466-8da1-eb3c861a59dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:16.148451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.148421 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-token" pod "3f43a283-0121-4466-8da1-eb3c861a59dd" (UID: "3f43a283-0121-4466-8da1-eb3c861a59dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:16.149339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.149317 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-envoy" pod "3f43a283-0121-4466-8da1-eb3c861a59dd" (UID: "3f43a283-0121-4466-8da1-eb3c861a59dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:16.154036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.153850 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-podinfo" pod "3f43a283-0121-4466-8da1-eb3c861a59dd" (UID: "3f43a283-0121-4466-8da1-eb3c861a59dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:21:16.156343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.156322 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-kube-api-access-7kc9d" pod "3f43a283-0121-4466-8da1-eb3c861a59dd" (UID: "3f43a283-0121-4466-8da1-eb3c861a59dd"). InnerVolumeSpecName "kube-api-access-7kc9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:16.248287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.248210 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.248287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.248240 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kc9d\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-kube-api-access-7kc9d\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.248287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.248250 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.248287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.248259 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f43a283-0121-4466-8da1-eb3c861a59dd-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.248287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.248268 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.248287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.248276 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.248287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.248283 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f43a283-0121-4466-8da1-eb3c861a59dd-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:16.335639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.335606 5736 generic.go:356] "Generic (PLEG): container finished" podID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerID="a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400" exitCode=0 Oct 29 23:21:16.335828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.335664 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" Oct 29 23:21:16.335828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.335702 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" event={"ID":"3f43a283-0121-4466-8da1-eb3c861a59dd","Type":"ContainerDied","Data":"a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400"} Oct 29 23:21:16.335828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.335770 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb" event={"ID":"3f43a283-0121-4466-8da1-eb3c861a59dd","Type":"ContainerDied","Data":"e49f9a7c3913f717fdfd742b6e81b002022d523d4c59e198cac657b54bf77eb3"} Oct 29 23:21:16.335828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.335810 5736 scope.go:122] "RemoveContainer" containerID="a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400" Oct 29 23:21:16.348512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.348497 5736 scope.go:122] "RemoveContainer" containerID="a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400" Oct 29 23:21:16.348820 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:21:16.348799 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400\": container with ID starting with a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400 not found: ID does not exist" containerID="a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400" Oct 29 23:21:16.348888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.348828 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400"} err="failed to get container status \"a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400\": rpc error: code = NotFound desc = could not find container \"a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400\": container with ID starting with a98ba900bd2bebdfefa060d75f630e205c4572fed1213e26656da5b0206f9400 not found: ID does not exist" Oct 29 23:21:16.382242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.382216 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb"] Oct 29 23:21:16.391872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:16.391850 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-jtfnb"] Oct 29 23:21:17.005488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:17.005452 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" path="/var/lib/kubelet/pods/3f43a283-0121-4466-8da1-eb3c861a59dd/volumes" Oct 29 23:21:19.862209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.862174 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr"] Oct 29 23:21:19.862996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.862975 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerName="istio-proxy" Oct 29 23:21:19.862996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.862996 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerName="istio-proxy" Oct 29 23:21:19.863122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.863117 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f43a283-0121-4466-8da1-eb3c861a59dd" containerName="istio-proxy" Oct 29 23:21:19.866460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.866442 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:19.878952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.878930 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:21:19.889003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.888982 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:21:19.894049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.894024 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr"] Oct 29 23:21:19.896684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.896660 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlvz\" (UniqueName: \"kubernetes.io/projected/98ec3f89-7376-4f75-a117-cd175ede6ccb-kube-api-access-qjlvz\") pod \"maas-api-revoke-sub-ff6b35a3-ttqlr\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:19.896833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.896811 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff6b35a3-ttqlr\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:19.896903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.896853 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff6b35a3-ttqlr\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:19.997840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.997811 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff6b35a3-ttqlr\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:19.998007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.997849 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff6b35a3-ttqlr\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:19.998090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.998015 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlvz\" (UniqueName: \"kubernetes.io/projected/98ec3f89-7376-4f75-a117-cd175ede6ccb-kube-api-access-qjlvz\") pod \"maas-api-revoke-sub-ff6b35a3-ttqlr\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:19.998505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:19.998482 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff6b35a3-ttqlr\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:20.000336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:20.000313 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff6b35a3-ttqlr\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:20.019552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:20.019526 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlvz\" (UniqueName: \"kubernetes.io/projected/98ec3f89-7376-4f75-a117-cd175ede6ccb-kube-api-access-qjlvz\") pod \"maas-api-revoke-sub-ff6b35a3-ttqlr\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:20.180721 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:20.180647 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:20.493665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:20.493639 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr"] Oct 29 23:21:20.496411 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:20.496386 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ec3f89_7376_4f75_a117_cd175ede6ccb.slice/crio-714b26501cd35031d2319881d52b68d14caab2dbc73f028a71f5faa9dd716687 WatchSource:0}: Error finding container 714b26501cd35031d2319881d52b68d14caab2dbc73f028a71f5faa9dd716687: Status 404 returned error can't find the container with id 714b26501cd35031d2319881d52b68d14caab2dbc73f028a71f5faa9dd716687 Oct 29 23:21:21.376289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:21.376247 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" event={"ID":"98ec3f89-7376-4f75-a117-cd175ede6ccb","Type":"ContainerStarted","Data":"6451a1105055f081c8fe0104ef1e88ffb6a4d116fbecb0bc8370917089470e08"} Oct 29 23:21:21.376695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:21.376308 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" event={"ID":"98ec3f89-7376-4f75-a117-cd175ede6ccb","Type":"ContainerStarted","Data":"714b26501cd35031d2319881d52b68d14caab2dbc73f028a71f5faa9dd716687"} Oct 29 23:21:21.413711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:21.413654 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" podStartSLOduration=2.413640123 podStartE2EDuration="2.413640123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:21.411828848 +0000 UTC m=+2278.527748455" watchObservedRunningTime="2026-10-29 23:21:21.413640123 +0000 UTC m=+2278.529559729" Oct 29 23:21:22.384752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:22.384718 5736 generic.go:356] "Generic (PLEG): container finished" podID="98ec3f89-7376-4f75-a117-cd175ede6ccb" containerID="6451a1105055f081c8fe0104ef1e88ffb6a4d116fbecb0bc8370917089470e08" exitCode=0 Oct 29 23:21:22.385145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:22.384795 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" event={"ID":"98ec3f89-7376-4f75-a117-cd175ede6ccb","Type":"ContainerDied","Data":"6451a1105055f081c8fe0104ef1e88ffb6a4d116fbecb0bc8370917089470e08"} Oct 29 23:21:23.553871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.553847 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:23.677764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.677684 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ec3f89-7376-4f75-a117-cd175ede6ccb-kube-api-access-qjlvz\" (UniqueName: \"kubernetes.io/projected/98ec3f89-7376-4f75-a117-cd175ede6ccb-kube-api-access-qjlvz\") pod \"98ec3f89-7376-4f75-a117-cd175ede6ccb\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " Oct 29 23:21:23.677943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.677772 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-openshift-ca\") pod \"98ec3f89-7376-4f75-a117-cd175ede6ccb\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " Oct 29 23:21:23.677943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.677814 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-xks-ca\") pod \"98ec3f89-7376-4f75-a117-cd175ede6ccb\" (UID: \"98ec3f89-7376-4f75-a117-cd175ede6ccb\") " Oct 29 23:21:23.678167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.678143 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-openshift-ca" pod "98ec3f89-7376-4f75-a117-cd175ede6ccb" (UID: "98ec3f89-7376-4f75-a117-cd175ede6ccb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:23.678430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.678404 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:23.679929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.679904 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ec3f89-7376-4f75-a117-cd175ede6ccb-kube-api-access-qjlvz" pod "98ec3f89-7376-4f75-a117-cd175ede6ccb" (UID: "98ec3f89-7376-4f75-a117-cd175ede6ccb"). InnerVolumeSpecName "kube-api-access-qjlvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:23.680063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.679997 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-xks-ca" pod "98ec3f89-7376-4f75-a117-cd175ede6ccb" (UID: "98ec3f89-7376-4f75-a117-cd175ede6ccb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:23.779440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.779415 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjlvz\" (UniqueName: \"kubernetes.io/projected/98ec3f89-7376-4f75-a117-cd175ede6ccb-kube-api-access-qjlvz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:23.779440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:23.779441 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ec3f89-7376-4f75-a117-cd175ede6ccb-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:24.425179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:24.425147 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" Oct 29 23:21:24.425383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:24.425340 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr" event={"ID":"98ec3f89-7376-4f75-a117-cd175ede6ccb","Type":"ContainerDied","Data":"714b26501cd35031d2319881d52b68d14caab2dbc73f028a71f5faa9dd716687"} Oct 29 23:21:24.425438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:24.425423 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="714b26501cd35031d2319881d52b68d14caab2dbc73f028a71f5faa9dd716687" Oct 29 23:21:25.331860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:25.331828 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:21:26.843631 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:21:26.840811 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:21:38.771528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.771485 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh"] Oct 29 23:21:38.772341 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.772317 5736 cpu_manager.go:414] "containerMap: removing container" podUID="98ec3f89-7376-4f75-a117-cd175ede6ccb" containerName="revoke-keys" Oct 29 23:21:38.772449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.772343 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ec3f89-7376-4f75-a117-cd175ede6ccb" containerName="revoke-keys" Oct 29 23:21:38.772566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.772551 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ec3f89-7376-4f75-a117-cd175ede6ccb" containerName="revoke-keys" Oct 29 23:21:38.777953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.777930 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:38.795908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.795872 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh"] Oct 29 23:21:38.802206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.802167 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:21:38.802303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.802208 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:21:38.847518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.847010 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad46e72-f065-4159-9270-28234a227d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81829674-bgnhh\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:38.847518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.847161 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad46e72-f065-4159-9270-28234a227d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81829674-bgnhh\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:38.847518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.847226 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rglw\" (UniqueName: \"kubernetes.io/projected/3ad46e72-f065-4159-9270-28234a227d85-kube-api-access-7rglw\") pod \"maas-api-revoke-sub-81829674-bgnhh\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:38.948441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.948405 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad46e72-f065-4159-9270-28234a227d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81829674-bgnhh\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:38.948663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.948467 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rglw\" (UniqueName: \"kubernetes.io/projected/3ad46e72-f065-4159-9270-28234a227d85-kube-api-access-7rglw\") pod \"maas-api-revoke-sub-81829674-bgnhh\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:38.948782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.948759 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad46e72-f065-4159-9270-28234a227d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81829674-bgnhh\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:38.949137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.949116 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad46e72-f065-4159-9270-28234a227d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81829674-bgnhh\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:38.951266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.951238 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad46e72-f065-4159-9270-28234a227d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81829674-bgnhh\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:38.977934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:38.977902 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rglw\" (UniqueName: \"kubernetes.io/projected/3ad46e72-f065-4159-9270-28234a227d85-kube-api-access-7rglw\") pod \"maas-api-revoke-sub-81829674-bgnhh\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:39.089480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.089400 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv"] Oct 29 23:21:39.091545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.091519 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:39.093750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.093731 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.121208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.118670 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv"] Oct 29 23:21:39.129669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.129641 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6"] Oct 29 23:21:39.137630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.135573 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.154135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.152395 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5wbz\" (UniqueName: \"kubernetes.io/projected/bbcec93b-efd9-4733-a44d-34e9c2ff907d-kube-api-access-m5wbz\") pod \"maas-api-revoke-sub-96c8daed-qsfqv\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.154135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.152458 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c8daed-qsfqv\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.154135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.152734 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c8daed-qsfqv\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.192763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.188675 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6"] Oct 29 23:21:39.255645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.255599 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2713814f-6jrw6\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.255843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.255706 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c8daed-qsfqv\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.255949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.255927 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2713814f-6jrw6\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.256013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.255977 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmdq\" (UniqueName: \"kubernetes.io/projected/6a634441-1a2f-4e89-a3ec-169c5db13f15-kube-api-access-8pmdq\") pod \"maas-api-revoke-sub-2713814f-6jrw6\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.256079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.256025 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5wbz\" (UniqueName: \"kubernetes.io/projected/bbcec93b-efd9-4733-a44d-34e9c2ff907d-kube-api-access-m5wbz\") pod \"maas-api-revoke-sub-96c8daed-qsfqv\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.256079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.256058 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c8daed-qsfqv\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.262287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.261474 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c8daed-qsfqv\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.262454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.262422 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c8daed-qsfqv\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.309605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.307147 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5wbz\" (UniqueName: \"kubernetes.io/projected/bbcec93b-efd9-4733-a44d-34e9c2ff907d-kube-api-access-m5wbz\") pod \"maas-api-revoke-sub-96c8daed-qsfqv\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.364651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.363629 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2713814f-6jrw6\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.364651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.363709 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmdq\" (UniqueName: \"kubernetes.io/projected/6a634441-1a2f-4e89-a3ec-169c5db13f15-kube-api-access-8pmdq\") pod \"maas-api-revoke-sub-2713814f-6jrw6\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.364651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.363854 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2713814f-6jrw6\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.392242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.392200 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2713814f-6jrw6\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.466828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.466795 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:39.494182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.493044 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh"] Oct 29 23:21:39.504736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.504667 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2713814f-6jrw6\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.507708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.507653 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmdq\" (UniqueName: \"kubernetes.io/projected/6a634441-1a2f-4e89-a3ec-169c5db13f15-kube-api-access-8pmdq\") pod \"maas-api-revoke-sub-2713814f-6jrw6\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.582176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.582147 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" event={"ID":"3ad46e72-f065-4159-9270-28234a227d85","Type":"ContainerStarted","Data":"01a17a1bc408a03506e87cbc579128dca75ccec20f1cb6d7e8d4803af15fb754"} Oct 29 23:21:39.801252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.801217 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:39.829769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:39.829739 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv"] Oct 29 23:21:39.831873 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:39.831842 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcec93b_efd9_4733_a44d_34e9c2ff907d.slice/crio-40ea4fb7b60c21a940604cb390792d537385fbf3cd6dd5f9a80c58f920fa0693 WatchSource:0}: Error finding container 40ea4fb7b60c21a940604cb390792d537385fbf3cd6dd5f9a80c58f920fa0693: Status 404 returned error can't find the container with id 40ea4fb7b60c21a940604cb390792d537385fbf3cd6dd5f9a80c58f920fa0693 Oct 29 23:21:40.126441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:40.126239 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6"] Oct 29 23:21:40.129518 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:40.129490 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a634441_1a2f_4e89_a3ec_169c5db13f15.slice/crio-4a66446eca22a562dc55d3f50600766efa5ba9093372a4a24511b4983b8cec83 WatchSource:0}: Error finding container 4a66446eca22a562dc55d3f50600766efa5ba9093372a4a24511b4983b8cec83: Status 404 returned error can't find the container with id 4a66446eca22a562dc55d3f50600766efa5ba9093372a4a24511b4983b8cec83 Oct 29 23:21:40.591957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:40.591918 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" event={"ID":"3ad46e72-f065-4159-9270-28234a227d85","Type":"ContainerStarted","Data":"2bd38b104d7fdf2b28fc3be39c968357a661e47d053eeb558b9ccef8be72f62a"} Oct 29 23:21:40.593771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:40.593740 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" event={"ID":"6a634441-1a2f-4e89-a3ec-169c5db13f15","Type":"ContainerStarted","Data":"caae1a0ffad3b25d67b8b4c4d86a9011c182e802d1b7171d835e23a228cae62a"} Oct 29 23:21:40.593904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:40.593790 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" event={"ID":"6a634441-1a2f-4e89-a3ec-169c5db13f15","Type":"ContainerStarted","Data":"4a66446eca22a562dc55d3f50600766efa5ba9093372a4a24511b4983b8cec83"} Oct 29 23:21:40.595469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:40.595448 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" event={"ID":"bbcec93b-efd9-4733-a44d-34e9c2ff907d","Type":"ContainerStarted","Data":"6880dc71ade6a687ed06b326a7c18343b06411766254f3b6d343df46964d9cfa"} Oct 29 23:21:40.595550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:40.595485 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" event={"ID":"bbcec93b-efd9-4733-a44d-34e9c2ff907d","Type":"ContainerStarted","Data":"40ea4fb7b60c21a940604cb390792d537385fbf3cd6dd5f9a80c58f920fa0693"} Oct 29 23:21:40.631189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:40.631107 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" podStartSLOduration=2.631094462 podStartE2EDuration="2.631094462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:40.627744697 +0000 UTC m=+2297.743664303" watchObservedRunningTime="2026-10-29 23:21:40.631094462 +0000 UTC m=+2297.747014079" Oct 29 23:21:40.663505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:40.663463 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" podStartSLOduration=1.6634514710000001 podStartE2EDuration="1.663451471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:40.663281248 +0000 UTC m=+2297.779200859" watchObservedRunningTime="2026-10-29 23:21:40.663451471 +0000 UTC m=+2297.779371077" Oct 29 23:21:40.699911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:40.699861 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" podStartSLOduration=1.699847919 podStartE2EDuration="1.699847919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:40.698703963 +0000 UTC m=+2297.814623570" watchObservedRunningTime="2026-10-29 23:21:40.699847919 +0000 UTC m=+2297.815767525" Oct 29 23:21:41.195599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:41.195553 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995"] Oct 29 23:21:41.196078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:41.195821 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" containerID="cri-o://3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a" gracePeriod=30 Oct 29 23:21:41.604840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:41.604806 5736 generic.go:356] "Generic (PLEG): container finished" podID="3ad46e72-f065-4159-9270-28234a227d85" containerID="2bd38b104d7fdf2b28fc3be39c968357a661e47d053eeb558b9ccef8be72f62a" exitCode=0 Oct 29 23:21:41.605119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:41.604884 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" event={"ID":"3ad46e72-f065-4159-9270-28234a227d85","Type":"ContainerDied","Data":"2bd38b104d7fdf2b28fc3be39c968357a661e47d053eeb558b9ccef8be72f62a"} Oct 29 23:21:41.606522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:41.606496 5736 generic.go:356] "Generic (PLEG): container finished" podID="6a634441-1a2f-4e89-a3ec-169c5db13f15" containerID="caae1a0ffad3b25d67b8b4c4d86a9011c182e802d1b7171d835e23a228cae62a" exitCode=0 Oct 29 23:21:41.606669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:41.606524 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" event={"ID":"6a634441-1a2f-4e89-a3ec-169c5db13f15","Type":"ContainerDied","Data":"caae1a0ffad3b25d67b8b4c4d86a9011c182e802d1b7171d835e23a228cae62a"} Oct 29 23:21:41.608196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:41.608175 5736 generic.go:356] "Generic (PLEG): container finished" podID="bbcec93b-efd9-4733-a44d-34e9c2ff907d" containerID="6880dc71ade6a687ed06b326a7c18343b06411766254f3b6d343df46964d9cfa" exitCode=0 Oct 29 23:21:41.608316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:41.608259 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" event={"ID":"bbcec93b-efd9-4733-a44d-34e9c2ff907d","Type":"ContainerDied","Data":"6880dc71ade6a687ed06b326a7c18343b06411766254f3b6d343df46964d9cfa"} Oct 29 23:21:42.810356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.810328 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:42.877287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.877257 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:42.896696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.896671 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:42.963843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.963686 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-openshift-ca\") pod \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " Oct 29 23:21:42.971017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.964130 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-openshift-ca" pod "bbcec93b-efd9-4733-a44d-34e9c2ff907d" (UID: "bbcec93b-efd9-4733-a44d-34e9c2ff907d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:42.971017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.966647 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcec93b-efd9-4733-a44d-34e9c2ff907d-kube-api-access-m5wbz\" (UniqueName: \"kubernetes.io/projected/bbcec93b-efd9-4733-a44d-34e9c2ff907d-kube-api-access-m5wbz\") pod \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " Oct 29 23:21:42.971017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.966948 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-xks-ca\") pod \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\" (UID: \"bbcec93b-efd9-4733-a44d-34e9c2ff907d\") " Oct 29 23:21:42.971017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.970285 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcec93b-efd9-4733-a44d-34e9c2ff907d-kube-api-access-m5wbz" pod "bbcec93b-efd9-4733-a44d-34e9c2ff907d" (UID: "bbcec93b-efd9-4733-a44d-34e9c2ff907d"). InnerVolumeSpecName "kube-api-access-m5wbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:42.971017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.970554 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx"] Oct 29 23:21:42.971017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.970690 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-xks-ca" pod "bbcec93b-efd9-4733-a44d-34e9c2ff907d" (UID: "bbcec93b-efd9-4733-a44d-34e9c2ff907d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:42.971441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.971420 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3ad46e72-f065-4159-9270-28234a227d85" containerName="revoke-keys" Oct 29 23:21:42.971518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.971443 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad46e72-f065-4159-9270-28234a227d85" containerName="revoke-keys" Oct 29 23:21:42.971518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.971476 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bbcec93b-efd9-4733-a44d-34e9c2ff907d" containerName="revoke-keys" Oct 29 23:21:42.971518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.971484 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcec93b-efd9-4733-a44d-34e9c2ff907d" containerName="revoke-keys" Oct 29 23:21:42.971706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.971550 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6a634441-1a2f-4e89-a3ec-169c5db13f15" containerName="revoke-keys" Oct 29 23:21:42.971706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.971559 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a634441-1a2f-4e89-a3ec-169c5db13f15" containerName="revoke-keys" Oct 29 23:21:42.971816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.971760 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a634441-1a2f-4e89-a3ec-169c5db13f15" containerName="revoke-keys" Oct 29 23:21:42.971816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.971779 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad46e72-f065-4159-9270-28234a227d85" containerName="revoke-keys" Oct 29 23:21:42.971816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.971793 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcec93b-efd9-4733-a44d-34e9c2ff907d" containerName="revoke-keys" Oct 29 23:21:42.975443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.975421 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:42.989706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:42.989651 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx"] Oct 29 23:21:43.068966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.068929 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-xks-ca\") pod \"6a634441-1a2f-4e89-a3ec-169c5db13f15\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " Oct 29 23:21:43.069162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.068993 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ad46e72-f065-4159-9270-28234a227d85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad46e72-f065-4159-9270-28234a227d85-maas-api-xks-ca\") pod \"3ad46e72-f065-4159-9270-28234a227d85\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " Oct 29 23:21:43.069162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.069060 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad46e72-f065-4159-9270-28234a227d85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad46e72-f065-4159-9270-28234a227d85-maas-api-openshift-ca\") pod \"3ad46e72-f065-4159-9270-28234a227d85\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " Oct 29 23:21:43.069312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.069177 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad46e72-f065-4159-9270-28234a227d85-kube-api-access-7rglw\" (UniqueName: \"kubernetes.io/projected/3ad46e72-f065-4159-9270-28234a227d85-kube-api-access-7rglw\") pod \"3ad46e72-f065-4159-9270-28234a227d85\" (UID: \"3ad46e72-f065-4159-9270-28234a227d85\") " Oct 29 23:21:43.069312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.069241 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a634441-1a2f-4e89-a3ec-169c5db13f15-kube-api-access-8pmdq\" (UniqueName: \"kubernetes.io/projected/6a634441-1a2f-4e89-a3ec-169c5db13f15-kube-api-access-8pmdq\") pod \"6a634441-1a2f-4e89-a3ec-169c5db13f15\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " Oct 29 23:21:43.069422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.069324 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-openshift-ca\") pod \"6a634441-1a2f-4e89-a3ec-169c5db13f15\" (UID: \"6a634441-1a2f-4e89-a3ec-169c5db13f15\") " Oct 29 23:21:43.069537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.069503 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad46e72-f065-4159-9270-28234a227d85-maas-api-openshift-ca" pod "3ad46e72-f065-4159-9270-28234a227d85" (UID: "3ad46e72-f065-4159-9270-28234a227d85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:43.069869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.069844 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-openshift-ca" pod "6a634441-1a2f-4e89-a3ec-169c5db13f15" (UID: "6a634441-1a2f-4e89-a3ec-169c5db13f15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:43.070055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.070032 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:43.070157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.070061 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcec93b-efd9-4733-a44d-34e9c2ff907d-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:43.070157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.070075 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad46e72-f065-4159-9270-28234a227d85-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:43.070157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.070087 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5wbz\" (UniqueName: \"kubernetes.io/projected/bbcec93b-efd9-4733-a44d-34e9c2ff907d-kube-api-access-m5wbz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:43.070157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.070102 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:43.071408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.071386 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a634441-1a2f-4e89-a3ec-169c5db13f15-kube-api-access-8pmdq" pod "6a634441-1a2f-4e89-a3ec-169c5db13f15" (UID: "6a634441-1a2f-4e89-a3ec-169c5db13f15"). InnerVolumeSpecName "kube-api-access-8pmdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:43.071499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.071425 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad46e72-f065-4159-9270-28234a227d85-kube-api-access-7rglw" pod "3ad46e72-f065-4159-9270-28234a227d85" (UID: "3ad46e72-f065-4159-9270-28234a227d85"). InnerVolumeSpecName "kube-api-access-7rglw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:43.071649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.071632 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ad46e72-f065-4159-9270-28234a227d85-maas-api-xks-ca" pod "3ad46e72-f065-4159-9270-28234a227d85" (UID: "3ad46e72-f065-4159-9270-28234a227d85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:43.071806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.071784 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-xks-ca" pod "6a634441-1a2f-4e89-a3ec-169c5db13f15" (UID: "6a634441-1a2f-4e89-a3ec-169c5db13f15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:43.171859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.171783 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhtf\" (UniqueName: \"kubernetes.io/projected/f442c216-a87a-4e8f-baf8-73639be29834-kube-api-access-xkhtf\") pod \"maas-api-revoke-sub-275251cf-xd5kx\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.171859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.171846 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-275251cf-xd5kx\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.172102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.172078 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-275251cf-xd5kx\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.172440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.172419 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a634441-1a2f-4e89-a3ec-169c5db13f15-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:43.172485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.172446 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad46e72-f065-4159-9270-28234a227d85-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:43.172485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.172462 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rglw\" (UniqueName: \"kubernetes.io/projected/3ad46e72-f065-4159-9270-28234a227d85-kube-api-access-7rglw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:43.172485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.172478 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmdq\" (UniqueName: \"kubernetes.io/projected/6a634441-1a2f-4e89-a3ec-169c5db13f15-kube-api-access-8pmdq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:43.274059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.274028 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhtf\" (UniqueName: \"kubernetes.io/projected/f442c216-a87a-4e8f-baf8-73639be29834-kube-api-access-xkhtf\") pod \"maas-api-revoke-sub-275251cf-xd5kx\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.274218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.274082 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-275251cf-xd5kx\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.274218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.274111 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-275251cf-xd5kx\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.274788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.274767 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-275251cf-xd5kx\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.276163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.276143 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-275251cf-xd5kx\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.295471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.295445 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhtf\" (UniqueName: \"kubernetes.io/projected/f442c216-a87a-4e8f-baf8-73639be29834-kube-api-access-xkhtf\") pod \"maas-api-revoke-sub-275251cf-xd5kx\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.296219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.296202 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:43.606977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.606892 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx"] Oct 29 23:21:43.609765 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:43.609736 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf442c216_a87a_4e8f_baf8_73639be29834.slice/crio-41be993b2ca00d2f86a2e23538ea4a59d56152dd9fd0d558095386299f66d1cd WatchSource:0}: Error finding container 41be993b2ca00d2f86a2e23538ea4a59d56152dd9fd0d558095386299f66d1cd: Status 404 returned error can't find the container with id 41be993b2ca00d2f86a2e23538ea4a59d56152dd9fd0d558095386299f66d1cd Oct 29 23:21:43.627288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.627265 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" Oct 29 23:21:43.627422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.627305 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh" event={"ID":"3ad46e72-f065-4159-9270-28234a227d85","Type":"ContainerDied","Data":"01a17a1bc408a03506e87cbc579128dca75ccec20f1cb6d7e8d4803af15fb754"} Oct 29 23:21:43.627422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.627350 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01a17a1bc408a03506e87cbc579128dca75ccec20f1cb6d7e8d4803af15fb754" Oct 29 23:21:43.629019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.628992 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" event={"ID":"f442c216-a87a-4e8f-baf8-73639be29834","Type":"ContainerStarted","Data":"41be993b2ca00d2f86a2e23538ea4a59d56152dd9fd0d558095386299f66d1cd"} Oct 29 23:21:43.630612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.630590 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" Oct 29 23:21:43.630746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.630607 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6" event={"ID":"6a634441-1a2f-4e89-a3ec-169c5db13f15","Type":"ContainerDied","Data":"4a66446eca22a562dc55d3f50600766efa5ba9093372a4a24511b4983b8cec83"} Oct 29 23:21:43.630746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.630651 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a66446eca22a562dc55d3f50600766efa5ba9093372a4a24511b4983b8cec83" Oct 29 23:21:43.632266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.632240 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" event={"ID":"bbcec93b-efd9-4733-a44d-34e9c2ff907d","Type":"ContainerDied","Data":"40ea4fb7b60c21a940604cb390792d537385fbf3cd6dd5f9a80c58f920fa0693"} Oct 29 23:21:43.632373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.632280 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv" Oct 29 23:21:43.632442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:43.632284 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ea4fb7b60c21a940604cb390792d537385fbf3cd6dd5f9a80c58f920fa0693" Oct 29 23:21:44.642040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:44.642001 5736 generic.go:356] "Generic (PLEG): container finished" podID="f442c216-a87a-4e8f-baf8-73639be29834" containerID="a144d0e99ce2a28e819abc4174837bed3ca8f93141efcce2bc17d45c2a5b7ccd" exitCode=0 Oct 29 23:21:44.642426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:44.642083 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" event={"ID":"f442c216-a87a-4e8f-baf8-73639be29834","Type":"ContainerDied","Data":"a144d0e99ce2a28e819abc4174837bed3ca8f93141efcce2bc17d45c2a5b7ccd"} Oct 29 23:21:45.728297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.727904 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:21:45.728297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.727966 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:21:45.806181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.806156 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:45.912875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.912787 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f442c216-a87a-4e8f-baf8-73639be29834-kube-api-access-xkhtf\" (UniqueName: \"kubernetes.io/projected/f442c216-a87a-4e8f-baf8-73639be29834-kube-api-access-xkhtf\") pod \"f442c216-a87a-4e8f-baf8-73639be29834\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " Oct 29 23:21:45.913065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.913047 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-xks-ca\") pod \"f442c216-a87a-4e8f-baf8-73639be29834\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " Oct 29 23:21:45.913144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.913087 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-openshift-ca\") pod \"f442c216-a87a-4e8f-baf8-73639be29834\" (UID: \"f442c216-a87a-4e8f-baf8-73639be29834\") " Oct 29 23:21:45.913489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.913461 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-openshift-ca" pod "f442c216-a87a-4e8f-baf8-73639be29834" (UID: "f442c216-a87a-4e8f-baf8-73639be29834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:45.913693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.913674 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:45.914966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.914943 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f442c216-a87a-4e8f-baf8-73639be29834-kube-api-access-xkhtf" pod "f442c216-a87a-4e8f-baf8-73639be29834" (UID: "f442c216-a87a-4e8f-baf8-73639be29834"). InnerVolumeSpecName "kube-api-access-xkhtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:45.915205 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:45.915186 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-xks-ca" pod "f442c216-a87a-4e8f-baf8-73639be29834" (UID: "f442c216-a87a-4e8f-baf8-73639be29834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:46.015404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.015371 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f442c216-a87a-4e8f-baf8-73639be29834-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.015404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.015403 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhtf\" (UniqueName: \"kubernetes.io/projected/f442c216-a87a-4e8f-baf8-73639be29834-kube-api-access-xkhtf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.466762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.466699 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 23:21:46.523018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.522983 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-token\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-token\") pod \"210b9778-d69a-4264-bcd5-1d13dfb17640\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " Oct 29 23:21:46.523184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.523141 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-certs\") pod \"210b9778-d69a-4264-bcd5-1d13dfb17640\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " Oct 29 23:21:46.523254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.523204 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-credential-socket\") pod \"210b9778-d69a-4264-bcd5-1d13dfb17640\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " Oct 29 23:21:46.523317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.523257 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-socket\") pod \"210b9778-d69a-4264-bcd5-1d13dfb17640\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " Oct 29 23:21:46.523381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.523338 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-kube-api-access-2dnxm\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-kube-api-access-2dnxm\") pod \"210b9778-d69a-4264-bcd5-1d13dfb17640\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " Oct 29 23:21:46.523447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.523378 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210b9778-d69a-4264-bcd5-1d13dfb17640-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9778-d69a-4264-bcd5-1d13dfb17640-istiod-ca-cert\") pod \"210b9778-d69a-4264-bcd5-1d13dfb17640\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " Oct 29 23:21:46.523503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.523459 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-podinfo\") pod \"210b9778-d69a-4264-bcd5-1d13dfb17640\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " Oct 29 23:21:46.523503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.523435 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-certs" pod "210b9778-d69a-4264-bcd5-1d13dfb17640" (UID: "210b9778-d69a-4264-bcd5-1d13dfb17640"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:46.523503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.523485 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-data\") pod \"210b9778-d69a-4264-bcd5-1d13dfb17640\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " Oct 29 23:21:46.523662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.523570 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-envoy\") pod \"210b9778-d69a-4264-bcd5-1d13dfb17640\" (UID: \"210b9778-d69a-4264-bcd5-1d13dfb17640\") " Oct 29 23:21:46.524693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.524668 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.529634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.525794 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-socket" pod "210b9778-d69a-4264-bcd5-1d13dfb17640" (UID: "210b9778-d69a-4264-bcd5-1d13dfb17640"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:46.529634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.525802 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-credential-socket" pod "210b9778-d69a-4264-bcd5-1d13dfb17640" (UID: "210b9778-d69a-4264-bcd5-1d13dfb17640"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:46.529634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.526126 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-data" pod "210b9778-d69a-4264-bcd5-1d13dfb17640" (UID: "210b9778-d69a-4264-bcd5-1d13dfb17640"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:46.529634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.526279 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210b9778-d69a-4264-bcd5-1d13dfb17640-istiod-ca-cert" pod "210b9778-d69a-4264-bcd5-1d13dfb17640" (UID: "210b9778-d69a-4264-bcd5-1d13dfb17640"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:46.531925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.530450 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-podinfo" pod "210b9778-d69a-4264-bcd5-1d13dfb17640" (UID: "210b9778-d69a-4264-bcd5-1d13dfb17640"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:21:46.531925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.530664 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-envoy" pod "210b9778-d69a-4264-bcd5-1d13dfb17640" (UID: "210b9778-d69a-4264-bcd5-1d13dfb17640"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:46.532259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.532230 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-token" pod "210b9778-d69a-4264-bcd5-1d13dfb17640" (UID: "210b9778-d69a-4264-bcd5-1d13dfb17640"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:46.533071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.533044 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-kube-api-access-2dnxm" pod "210b9778-d69a-4264-bcd5-1d13dfb17640" (UID: "210b9778-d69a-4264-bcd5-1d13dfb17640"). InnerVolumeSpecName "kube-api-access-2dnxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:46.626718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.626234 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.626718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.626269 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.626718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.626283 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.626718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.626297 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.626718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.626309 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.626718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.626322 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9778-d69a-4264-bcd5-1d13dfb17640-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.626718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.626335 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnxm\" (UniqueName: \"kubernetes.io/projected/210b9778-d69a-4264-bcd5-1d13dfb17640-kube-api-access-2dnxm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.626718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.626346 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9778-d69a-4264-bcd5-1d13dfb17640-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:46.658375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.658349 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" Oct 29 23:21:46.658516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.658353 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx" event={"ID":"f442c216-a87a-4e8f-baf8-73639be29834","Type":"ContainerDied","Data":"41be993b2ca00d2f86a2e23538ea4a59d56152dd9fd0d558095386299f66d1cd"} Oct 29 23:21:46.658516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.658437 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41be993b2ca00d2f86a2e23538ea4a59d56152dd9fd0d558095386299f66d1cd" Oct 29 23:21:46.660045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.660020 5736 generic.go:356] "Generic (PLEG): container finished" podID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerID="3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a" exitCode=0 Oct 29 23:21:46.660166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.660088 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" Oct 29 23:21:46.660166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.660102 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" event={"ID":"210b9778-d69a-4264-bcd5-1d13dfb17640","Type":"ContainerDied","Data":"3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a"} Oct 29 23:21:46.660166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.660157 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995" event={"ID":"210b9778-d69a-4264-bcd5-1d13dfb17640","Type":"ContainerDied","Data":"d745ee6c828b105f7ca73275e9e554d1c2a9f8e2f839bc0cbaac982a315facd6"} Oct 29 23:21:46.660307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.660184 5736 scope.go:122] "RemoveContainer" containerID="3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a" Oct 29 23:21:46.671462 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.671300 5736 scope.go:122] "RemoveContainer" containerID="3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a" Oct 29 23:21:46.671695 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:21:46.671660 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a\": container with ID starting with 3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a not found: ID does not exist" containerID="3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a" Oct 29 23:21:46.671770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.671704 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a"} err="failed to get container status \"3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a\": rpc error: code = NotFound desc = could not find container \"3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a\": container with ID starting with 3adb0f061028d4314f4005b8a58fcad7ecc4a62562152f7910da5f50bbc6a75a not found: ID does not exist" Oct 29 23:21:46.707794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.707770 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995"] Oct 29 23:21:46.715714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:46.715694 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rb995"] Oct 29 23:21:47.005365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:47.005335 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" path="/var/lib/kubelet/pods/210b9778-d69a-4264-bcd5-1d13dfb17640/volumes" Oct 29 23:21:49.131905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.131865 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr"] Oct 29 23:21:49.132681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.132659 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f442c216-a87a-4e8f-baf8-73639be29834" containerName="revoke-keys" Oct 29 23:21:49.132681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.132681 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f442c216-a87a-4e8f-baf8-73639be29834" containerName="revoke-keys" Oct 29 23:21:49.132869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.132714 5736 cpu_manager.go:414] "containerMap: removing container" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" Oct 29 23:21:49.132869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.132722 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" Oct 29 23:21:49.132982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.132939 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f442c216-a87a-4e8f-baf8-73639be29834" containerName="revoke-keys" Oct 29 23:21:49.132982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.132961 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="210b9778-d69a-4264-bcd5-1d13dfb17640" containerName="istio-proxy" Oct 29 23:21:49.136818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.136798 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.156757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.156728 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr"] Oct 29 23:21:49.170421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.170393 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:21:49.170723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.170706 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:21:49.272119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.272087 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fce0643-m88wr\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.272287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.272247 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkzhq\" (UniqueName: \"kubernetes.io/projected/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-kube-api-access-mkzhq\") pod \"maas-api-revoke-sub-5fce0643-m88wr\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.272424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.272403 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fce0643-m88wr\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.373421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.373391 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fce0643-m88wr\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.373592 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.373478 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fce0643-m88wr\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.373650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.373615 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkzhq\" (UniqueName: \"kubernetes.io/projected/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-kube-api-access-mkzhq\") pod \"maas-api-revoke-sub-5fce0643-m88wr\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.374205 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.374178 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fce0643-m88wr\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.375945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.375925 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fce0643-m88wr\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.395258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.395206 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkzhq\" (UniqueName: \"kubernetes.io/projected/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-kube-api-access-mkzhq\") pod \"maas-api-revoke-sub-5fce0643-m88wr\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.451780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.451756 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:49.525929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.525897 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss"] Oct 29 23:21:49.532163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.532140 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.554063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.550441 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss"] Oct 29 23:21:49.678251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.678170 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd6930f-lvbss\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.678411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.678270 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd6930f-lvbss\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.678483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.678459 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7896\" (UniqueName: \"kubernetes.io/projected/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-kube-api-access-f7896\") pod \"maas-api-revoke-sub-cbd6930f-lvbss\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.696030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.696003 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4"] Oct 29 23:21:49.700201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.700180 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:49.720139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.720112 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4"] Oct 29 23:21:49.762503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.762472 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr"] Oct 29 23:21:49.766461 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:49.766436 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0a6fb0_fa9c_415a_8cc1_fd7abcbf1076.slice/crio-22accbe6ce9dae4aa417b2991199fda17d6eae4ea3dd934d67655d801129ded1 WatchSource:0}: Error finding container 22accbe6ce9dae4aa417b2991199fda17d6eae4ea3dd934d67655d801129ded1: Status 404 returned error can't find the container with id 22accbe6ce9dae4aa417b2991199fda17d6eae4ea3dd934d67655d801129ded1 Oct 29 23:21:49.780127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.780104 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd6930f-lvbss\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.780216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.780142 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd6930f-lvbss\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.780216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.780167 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8895b4-qzdm4\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:49.780331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.780238 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8895b4-qzdm4\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:49.780400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.780320 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7896\" (UniqueName: \"kubernetes.io/projected/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-kube-api-access-f7896\") pod \"maas-api-revoke-sub-cbd6930f-lvbss\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.780490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.780450 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5lq\" (UniqueName: \"kubernetes.io/projected/28962275-e0e4-420f-917f-c4e4115fa30a-kube-api-access-qb5lq\") pod \"maas-api-revoke-sub-ed8895b4-qzdm4\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:49.780950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.780925 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd6930f-lvbss\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.782872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.782853 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd6930f-lvbss\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.801241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.801220 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7896\" (UniqueName: \"kubernetes.io/projected/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-kube-api-access-f7896\") pod \"maas-api-revoke-sub-cbd6930f-lvbss\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.851483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.851462 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:49.881545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.881516 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8895b4-qzdm4\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:49.881741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.881630 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8895b4-qzdm4\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:49.881741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.881721 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5lq\" (UniqueName: \"kubernetes.io/projected/28962275-e0e4-420f-917f-c4e4115fa30a-kube-api-access-qb5lq\") pod \"maas-api-revoke-sub-ed8895b4-qzdm4\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:49.882193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.882169 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8895b4-qzdm4\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:49.883841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.883813 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8895b4-qzdm4\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:49.903282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:49.903248 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5lq\" (UniqueName: \"kubernetes.io/projected/28962275-e0e4-420f-917f-c4e4115fa30a-kube-api-access-qb5lq\") pod \"maas-api-revoke-sub-ed8895b4-qzdm4\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:50.014271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.014232 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:50.166020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.165918 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss"] Oct 29 23:21:50.168834 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:50.168810 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d54fa65_aaf1_4479_8dd8_e3d24fd7d8ff.slice/crio-aee15309718afb5879ab2823a5603a98f7e0ab67ae7075ff8f2f426cacca2e72 WatchSource:0}: Error finding container aee15309718afb5879ab2823a5603a98f7e0ab67ae7075ff8f2f426cacca2e72: Status 404 returned error can't find the container with id aee15309718afb5879ab2823a5603a98f7e0ab67ae7075ff8f2f426cacca2e72 Oct 29 23:21:50.332885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.332205 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4"] Oct 29 23:21:50.335454 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:50.335420 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28962275_e0e4_420f_917f_c4e4115fa30a.slice/crio-bd480e4d80876f39e810584c256544dceaae803dc35163ec1d80e45bf2506837 WatchSource:0}: Error finding container bd480e4d80876f39e810584c256544dceaae803dc35163ec1d80e45bf2506837: Status 404 returned error can't find the container with id bd480e4d80876f39e810584c256544dceaae803dc35163ec1d80e45bf2506837 Oct 29 23:21:50.693119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.693030 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" event={"ID":"28962275-e0e4-420f-917f-c4e4115fa30a","Type":"ContainerStarted","Data":"bd480e4d80876f39e810584c256544dceaae803dc35163ec1d80e45bf2506837"} Oct 29 23:21:50.694782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.694751 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" event={"ID":"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff","Type":"ContainerStarted","Data":"5f3ac1aeff1e8546f652246be9fce6c76b8292eba02dd80311264daa3231d907"} Oct 29 23:21:50.694929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.694802 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" event={"ID":"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff","Type":"ContainerStarted","Data":"aee15309718afb5879ab2823a5603a98f7e0ab67ae7075ff8f2f426cacca2e72"} Oct 29 23:21:50.696324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.696297 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" event={"ID":"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076","Type":"ContainerStarted","Data":"5a1c7fecdc81bb65bf11c28a40619d3081d4d64d1f342af0c6ea512c2068fb84"} Oct 29 23:21:50.696413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.696340 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" event={"ID":"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076","Type":"ContainerStarted","Data":"22accbe6ce9dae4aa417b2991199fda17d6eae4ea3dd934d67655d801129ded1"} Oct 29 23:21:50.731863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.731812 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" podStartSLOduration=1.731400608 podStartE2EDuration="1.731400608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:50.730171172 +0000 UTC m=+2307.846090778" watchObservedRunningTime="2026-10-29 23:21:50.731400608 +0000 UTC m=+2307.847320216" Oct 29 23:21:50.767766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:50.767712 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" podStartSLOduration=1.767695161 podStartE2EDuration="1.767695161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:50.765790234 +0000 UTC m=+2307.881709843" watchObservedRunningTime="2026-10-29 23:21:50.767695161 +0000 UTC m=+2307.883614767" Oct 29 23:21:51.705586 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:51.705490 5736 generic.go:356] "Generic (PLEG): container finished" podID="3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076" containerID="5a1c7fecdc81bb65bf11c28a40619d3081d4d64d1f342af0c6ea512c2068fb84" exitCode=0 Oct 29 23:21:51.705962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:51.705590 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" event={"ID":"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076","Type":"ContainerDied","Data":"5a1c7fecdc81bb65bf11c28a40619d3081d4d64d1f342af0c6ea512c2068fb84"} Oct 29 23:21:51.707317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:51.707292 5736 generic.go:356] "Generic (PLEG): container finished" podID="28962275-e0e4-420f-917f-c4e4115fa30a" containerID="1b21ba24f48e257f7866b3458d62bf5e7fcd48abe17449819279afb05ad106f8" exitCode=0 Oct 29 23:21:51.707434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:51.707373 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" event={"ID":"28962275-e0e4-420f-917f-c4e4115fa30a","Type":"ContainerDied","Data":"1b21ba24f48e257f7866b3458d62bf5e7fcd48abe17449819279afb05ad106f8"} Oct 29 23:21:51.708842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:51.708818 5736 generic.go:356] "Generic (PLEG): container finished" podID="3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff" containerID="5f3ac1aeff1e8546f652246be9fce6c76b8292eba02dd80311264daa3231d907" exitCode=0 Oct 29 23:21:51.708975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:51.708894 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" event={"ID":"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff","Type":"ContainerDied","Data":"5f3ac1aeff1e8546f652246be9fce6c76b8292eba02dd80311264daa3231d907"} Oct 29 23:21:52.897886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:52.897856 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:52.976551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:52.976490 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:52.979908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:52.979887 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:53.044017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.043982 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-xks-ca\") pod \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " Oct 29 23:21:53.044213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.044090 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-kube-api-access-mkzhq\" (UniqueName: \"kubernetes.io/projected/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-kube-api-access-mkzhq\") pod \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " Oct 29 23:21:53.044380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.044361 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-openshift-ca\") pod \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\" (UID: \"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076\") " Oct 29 23:21:53.044816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.044786 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-openshift-ca" pod "3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076" (UID: "3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:53.045054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.045031 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:53.046231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.046204 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-xks-ca" pod "3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076" (UID: "3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:53.046779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.046752 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-kube-api-access-mkzhq" pod "3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076" (UID: "3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076"). InnerVolumeSpecName "kube-api-access-mkzhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:53.146394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.146361 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-xks-ca\") pod \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " Oct 29 23:21:53.146604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.146467 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-openshift-ca\") pod \"28962275-e0e4-420f-917f-c4e4115fa30a\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " Oct 29 23:21:53.146675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.146643 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-kube-api-access-f7896\" (UniqueName: \"kubernetes.io/projected/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-kube-api-access-f7896\") pod \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " Oct 29 23:21:53.146804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.146780 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28962275-e0e4-420f-917f-c4e4115fa30a-kube-api-access-qb5lq\" (UniqueName: \"kubernetes.io/projected/28962275-e0e4-420f-917f-c4e4115fa30a-kube-api-access-qb5lq\") pod \"28962275-e0e4-420f-917f-c4e4115fa30a\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " Oct 29 23:21:53.146961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.146810 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-openshift-ca" pod "28962275-e0e4-420f-917f-c4e4115fa30a" (UID: "28962275-e0e4-420f-917f-c4e4115fa30a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:53.146961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.146827 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-xks-ca\") pod \"28962275-e0e4-420f-917f-c4e4115fa30a\" (UID: \"28962275-e0e4-420f-917f-c4e4115fa30a\") " Oct 29 23:21:53.146961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.146846 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-openshift-ca\") pod \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\" (UID: \"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff\") " Oct 29 23:21:53.148000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.147244 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-openshift-ca" pod "3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff" (UID: "3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:53.148000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.147956 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:53.148000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.147986 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:53.148171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.148010 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkzhq\" (UniqueName: \"kubernetes.io/projected/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076-kube-api-access-mkzhq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:53.148171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.148025 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:53.148728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.148708 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-xks-ca" pod "3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff" (UID: "3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:53.149108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.149092 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-kube-api-access-f7896" pod "3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff" (UID: "3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff"). InnerVolumeSpecName "kube-api-access-f7896". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:53.149219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.149204 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28962275-e0e4-420f-917f-c4e4115fa30a-kube-api-access-qb5lq" pod "28962275-e0e4-420f-917f-c4e4115fa30a" (UID: "28962275-e0e4-420f-917f-c4e4115fa30a"). InnerVolumeSpecName "kube-api-access-qb5lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:53.149263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.149235 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-xks-ca" pod "28962275-e0e4-420f-917f-c4e4115fa30a" (UID: "28962275-e0e4-420f-917f-c4e4115fa30a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:53.249777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.249699 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7896\" (UniqueName: \"kubernetes.io/projected/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-kube-api-access-f7896\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:53.249777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.249739 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb5lq\" (UniqueName: \"kubernetes.io/projected/28962275-e0e4-420f-917f-c4e4115fa30a-kube-api-access-qb5lq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:53.249777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.249750 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28962275-e0e4-420f-917f-c4e4115fa30a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:53.249777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.249759 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:53.414028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.413989 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8"] Oct 29 23:21:53.414953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.414932 5736 cpu_manager.go:414] "containerMap: removing container" podUID="28962275-e0e4-420f-917f-c4e4115fa30a" containerName="revoke-keys" Oct 29 23:21:53.414953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.414951 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28962275-e0e4-420f-917f-c4e4115fa30a" containerName="revoke-keys" Oct 29 23:21:53.415160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.414978 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076" containerName="revoke-keys" Oct 29 23:21:53.415160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.414986 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076" containerName="revoke-keys" Oct 29 23:21:53.415160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.415004 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff" containerName="revoke-keys" Oct 29 23:21:53.415160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.415012 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff" containerName="revoke-keys" Oct 29 23:21:53.415372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.415181 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="28962275-e0e4-420f-917f-c4e4115fa30a" containerName="revoke-keys" Oct 29 23:21:53.415372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.415197 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076" containerName="revoke-keys" Oct 29 23:21:53.415372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.415209 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff" containerName="revoke-keys" Oct 29 23:21:53.420457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.420436 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.433759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.433735 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8"] Oct 29 23:21:53.554058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.553963 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dccb0ef3-97df8\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.554058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.554008 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dccb0ef3-97df8\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.554294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.554157 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz72\" (UniqueName: \"kubernetes.io/projected/acf648c5-e3af-42e9-a9a9-c885f79afaf5-kube-api-access-7gz72\") pod \"maas-api-revoke-sub-dccb0ef3-97df8\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.656099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.656054 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz72\" (UniqueName: \"kubernetes.io/projected/acf648c5-e3af-42e9-a9a9-c885f79afaf5-kube-api-access-7gz72\") pod \"maas-api-revoke-sub-dccb0ef3-97df8\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.656372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.656349 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dccb0ef3-97df8\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.656476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.656385 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dccb0ef3-97df8\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.657127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.657095 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dccb0ef3-97df8\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.658995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.658971 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dccb0ef3-97df8\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.677748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.677720 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz72\" (UniqueName: \"kubernetes.io/projected/acf648c5-e3af-42e9-a9a9-c885f79afaf5-kube-api-access-7gz72\") pod \"maas-api-revoke-sub-dccb0ef3-97df8\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:53.725282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.725258 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" Oct 29 23:21:53.725425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.725301 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4" event={"ID":"28962275-e0e4-420f-917f-c4e4115fa30a","Type":"ContainerDied","Data":"bd480e4d80876f39e810584c256544dceaae803dc35163ec1d80e45bf2506837"} Oct 29 23:21:53.725425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.725377 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd480e4d80876f39e810584c256544dceaae803dc35163ec1d80e45bf2506837" Oct 29 23:21:53.726928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.726912 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" Oct 29 23:21:53.727054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.726923 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss" event={"ID":"3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff","Type":"ContainerDied","Data":"aee15309718afb5879ab2823a5603a98f7e0ab67ae7075ff8f2f426cacca2e72"} Oct 29 23:21:53.727054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.726968 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee15309718afb5879ab2823a5603a98f7e0ab67ae7075ff8f2f426cacca2e72" Oct 29 23:21:53.728442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.728418 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" event={"ID":"3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076","Type":"ContainerDied","Data":"22accbe6ce9dae4aa417b2991199fda17d6eae4ea3dd934d67655d801129ded1"} Oct 29 23:21:53.728442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.728438 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr" Oct 29 23:21:53.728570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.728464 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22accbe6ce9dae4aa417b2991199fda17d6eae4ea3dd934d67655d801129ded1" Oct 29 23:21:53.735101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:53.735080 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:54.042099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:54.042059 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8"] Oct 29 23:21:54.055625 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:54.055590 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf648c5_e3af_42e9_a9a9_c885f79afaf5.slice/crio-6fee1f6dbcbd52dba54af9414a05a00c4cf40262c5b4d0e3d06cac09fc829cda WatchSource:0}: Error finding container 6fee1f6dbcbd52dba54af9414a05a00c4cf40262c5b4d0e3d06cac09fc829cda: Status 404 returned error can't find the container with id 6fee1f6dbcbd52dba54af9414a05a00c4cf40262c5b4d0e3d06cac09fc829cda Oct 29 23:21:54.737271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:54.737239 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" event={"ID":"acf648c5-e3af-42e9-a9a9-c885f79afaf5","Type":"ContainerStarted","Data":"2421a1b92f63fce4c21ccc9ce5f50356f28646560c1b326b5069ef99a7c64ea3"} Oct 29 23:21:54.737445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:54.737289 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" event={"ID":"acf648c5-e3af-42e9-a9a9-c885f79afaf5","Type":"ContainerStarted","Data":"6fee1f6dbcbd52dba54af9414a05a00c4cf40262c5b4d0e3d06cac09fc829cda"} Oct 29 23:21:54.774566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:54.774504 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" podStartSLOduration=1.774490604 podStartE2EDuration="1.774490604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:54.772757124 +0000 UTC m=+2311.888676730" watchObservedRunningTime="2026-10-29 23:21:54.774490604 +0000 UTC m=+2311.890410209" Oct 29 23:21:55.746044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:55.746013 5736 generic.go:356] "Generic (PLEG): container finished" podID="acf648c5-e3af-42e9-a9a9-c885f79afaf5" containerID="2421a1b92f63fce4c21ccc9ce5f50356f28646560c1b326b5069ef99a7c64ea3" exitCode=0 Oct 29 23:21:55.746445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:55.746102 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" event={"ID":"acf648c5-e3af-42e9-a9a9-c885f79afaf5","Type":"ContainerDied","Data":"2421a1b92f63fce4c21ccc9ce5f50356f28646560c1b326b5069ef99a7c64ea3"} Oct 29 23:21:56.835530 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:21:56.835489 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:21:56.913761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:56.913737 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:57.010933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.010863 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf648c5-e3af-42e9-a9a9-c885f79afaf5-kube-api-access-7gz72\" (UniqueName: \"kubernetes.io/projected/acf648c5-e3af-42e9-a9a9-c885f79afaf5-kube-api-access-7gz72\") pod \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " Oct 29 23:21:57.010933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.010913 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-xks-ca\") pod \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " Oct 29 23:21:57.011119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.010988 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-openshift-ca\") pod \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\" (UID: \"acf648c5-e3af-42e9-a9a9-c885f79afaf5\") " Oct 29 23:21:57.011321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.011291 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-openshift-ca" pod "acf648c5-e3af-42e9-a9a9-c885f79afaf5" (UID: "acf648c5-e3af-42e9-a9a9-c885f79afaf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:57.011636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.011616 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:57.013108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.013079 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf648c5-e3af-42e9-a9a9-c885f79afaf5-kube-api-access-7gz72" pod "acf648c5-e3af-42e9-a9a9-c885f79afaf5" (UID: "acf648c5-e3af-42e9-a9a9-c885f79afaf5"). InnerVolumeSpecName "kube-api-access-7gz72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:57.013170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.013106 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-xks-ca" pod "acf648c5-e3af-42e9-a9a9-c885f79afaf5" (UID: "acf648c5-e3af-42e9-a9a9-c885f79afaf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:21:57.113840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.113793 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gz72\" (UniqueName: \"kubernetes.io/projected/acf648c5-e3af-42e9-a9a9-c885f79afaf5-kube-api-access-7gz72\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:57.113840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.113835 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf648c5-e3af-42e9-a9a9-c885f79afaf5-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:21:57.761496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.761464 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" Oct 29 23:21:57.761703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.761501 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8" event={"ID":"acf648c5-e3af-42e9-a9a9-c885f79afaf5","Type":"ContainerDied","Data":"6fee1f6dbcbd52dba54af9414a05a00c4cf40262c5b4d0e3d06cac09fc829cda"} Oct 29 23:21:57.761703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:57.761575 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fee1f6dbcbd52dba54af9414a05a00c4cf40262c5b4d0e3d06cac09fc829cda" Oct 29 23:21:58.634481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.634445 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk"] Oct 29 23:21:58.635219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.635192 5736 cpu_manager.go:414] "containerMap: removing container" podUID="acf648c5-e3af-42e9-a9a9-c885f79afaf5" containerName="revoke-keys" Oct 29 23:21:58.635219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.635212 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf648c5-e3af-42e9-a9a9-c885f79afaf5" containerName="revoke-keys" Oct 29 23:21:58.635361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.635345 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf648c5-e3af-42e9-a9a9-c885f79afaf5" containerName="revoke-keys" Oct 29 23:21:58.639242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.639221 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.653075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.653053 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4ad17e-openshift-default-dockercfg-pkm5r\"" Oct 29 23:21:58.668804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.668782 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk"] Oct 29 23:21:58.740202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.740166 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbgxg\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-kube-api-access-hbgxg\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.740344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.740222 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-token\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.740409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.740371 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b691c-2aff-4b07-b398-90d86a6f68db-istiod-ca-cert\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.740457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.740430 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-certs\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.740516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.740495 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-data\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.740651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.740631 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.740744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.740665 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-credential-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.740789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.740776 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-podinfo\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.740851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.740801 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-envoy\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842039 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-certs\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842079 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-data\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842112 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842135 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-credential-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842198 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-podinfo\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842228 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-envoy\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842287 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbgxg\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-kube-api-access-hbgxg\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842329 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-token\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842468 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b691c-2aff-4b07-b398-90d86a6f68db-istiod-ca-cert\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842480 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-certs\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842528 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-data\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842689 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.842743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.842737 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-credential-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.843290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.843269 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b691c-2aff-4b07-b398-90d86a6f68db-istiod-ca-cert\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.844610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.844571 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-envoy\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.844840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.844821 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-podinfo\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.866271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.866242 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-token\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.866406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.866386 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbgxg\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-kube-api-access-hbgxg\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:58.953343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:58.953275 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:21:59.317841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:59.316709 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk"] Oct 29 23:21:59.319812 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:21:59.319784 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7b691c_2aff_4b07_b398_90d86a6f68db.slice/crio-6b9850b1cda962f5c93eaf10ad7b51d047c36d9d76f39bd1cf8abda319853532 WatchSource:0}: Error finding container 6b9850b1cda962f5c93eaf10ad7b51d047c36d9d76f39bd1cf8abda319853532: Status 404 returned error can't find the container with id 6b9850b1cda962f5c93eaf10ad7b51d047c36d9d76f39bd1cf8abda319853532 Oct 29 23:21:59.322466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:59.322437 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:21:59.322541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:59.322501 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:21:59.322541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:59.322529 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:21:59.780736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:59.780691 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" event={"ID":"3e7b691c-2aff-4b07-b398-90d86a6f68db","Type":"ContainerStarted","Data":"16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97"} Oct 29 23:21:59.781251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:59.780764 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" event={"ID":"3e7b691c-2aff-4b07-b398-90d86a6f68db","Type":"ContainerStarted","Data":"6b9850b1cda962f5c93eaf10ad7b51d047c36d9d76f39bd1cf8abda319853532"} Oct 29 23:21:59.833861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:59.833790 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" podStartSLOduration=1.833772208 podStartE2EDuration="1.833772208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:59.830455935 +0000 UTC m=+2316.946375541" watchObservedRunningTime="2026-10-29 23:21:59.833772208 +0000 UTC m=+2316.949691816" Oct 29 23:21:59.964922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:21:59.964882 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:22:00.958016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:00.957974 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:00.958402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:00.958050 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 29 23:22:01.686797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:01.686735 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm"] Oct 29 23:22:01.958661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:01.955045 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:01.958661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:01.955162 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 29 23:22:02.129635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.129598 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj"] Oct 29 23:22:02.413067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.413027 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl"] Oct 29 23:22:02.463912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.463872 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.477519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.477489 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl"] Oct 29 23:22:02.486982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.486962 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c83b50-openshift-default-dockercfg-r28p9\"" Oct 29 23:22:02.559878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.559837 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-token\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.560184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.560158 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istiod-ca-cert\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.560329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.560314 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-podinfo\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.560488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.560470 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-credential-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.560679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.560644 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-data\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.560930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.560912 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncbrp\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-kube-api-access-ncbrp\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.561156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.561138 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-certs\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.561304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.561287 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-envoy\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.561421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.561407 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.662634 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-data\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.662804 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncbrp\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-kube-api-access-ncbrp\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.662930 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-certs\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.662974 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-envoy\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.663014 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.663067 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-token\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.663117 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istiod-ca-cert\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.663164 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-podinfo\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.663236 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-credential-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.663794 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-credential-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.664065 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-data\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.664661 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-certs\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.665516 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-socket\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.669394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.666209 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istiod-ca-cert\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.679424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.679396 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-envoy\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.683561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.683531 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-podinfo\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.700288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.700262 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncbrp\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-kube-api-access-ncbrp\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.713533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.713503 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm"] Oct 29 23:22:02.720604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.720554 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-token\") pod \"e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.854325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.854292 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:02.957112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.957077 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:02.957358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:02.957329 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 29 23:22:03.358885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:03.358832 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl"] Oct 29 23:22:03.366989 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:22:03.366954 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d0f97b_8df7_4c65_940e_bbf2c5c1e6a8.slice/crio-08c41c04faed863873bd415326603b5c76b3305a2eb6bbd468158b64aaf6413c WatchSource:0}: Error finding container 08c41c04faed863873bd415326603b5c76b3305a2eb6bbd468158b64aaf6413c: Status 404 returned error can't find the container with id 08c41c04faed863873bd415326603b5c76b3305a2eb6bbd468158b64aaf6413c Oct 29 23:22:03.370368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:03.370335 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:03.370463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:03.370406 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:03.370463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:03.370435 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:03.864412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:03.864366 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" event={"ID":"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8","Type":"ContainerStarted","Data":"c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba"} Oct 29 23:22:03.864631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:03.864498 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" event={"ID":"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8","Type":"ContainerStarted","Data":"08c41c04faed863873bd415326603b5c76b3305a2eb6bbd468158b64aaf6413c"} Oct 29 23:22:03.908662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:03.908609 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" podStartSLOduration=1.9085934340000001 podStartE2EDuration="1.908593434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:03.906502905 +0000 UTC m=+2321.022422512" watchObservedRunningTime="2026-10-29 23:22:03.908593434 +0000 UTC m=+2321.024513033" Oct 29 23:22:03.957046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:03.957008 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:03.957355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:03.957310 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 29 23:22:04.870999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:04.870051 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:04.870999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:04.870173 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 29 23:22:04.870999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:04.870216 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 29 23:22:05.043690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:05.022721 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:05.043690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:05.022788 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 29 23:22:05.858002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:05.857962 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 29 23:22:05.860845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:05.860814 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 29 23:22:06.073475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:06.073429 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:22:06.074259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:06.074232 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:22:06.075443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:06.075420 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:22:06.909603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:06.902853 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 29 23:22:06.909603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:06.907993 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 29 23:22:07.856604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:07.855932 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 29 23:22:07.856604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:07.855995 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 29 23:22:08.078376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:08.078332 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c"] Oct 29 23:22:08.855605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:08.854944 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 29 23:22:08.855605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:08.855022 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 29 23:22:09.855973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:09.855936 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 29 23:22:09.856377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:09.855992 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 29 23:22:10.860573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:10.860539 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:10.861080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:10.861055 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:10.862092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:10.862068 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:22:11.298561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:11.298524 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm"] Oct 29 23:22:16.726558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:16.726510 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" podUID="1c9dfbdd-401e-4f49-bb40-176392c82957" containerName="main" containerID="cri-o://1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d" gracePeriod=45 Oct 29 23:22:16.856650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:16.856571 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" podUID="1c9dfbdd-401e-4f49-bb40-176392c82957" containerName="main" probeResult="failure" output="Get \"https://10.134.0.235:8000/ready\": dial tcp 10.134.0.235:8000: connect: connection refused" Oct 29 23:22:17.028178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.028148 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:22:17.075672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.075639 5736 generic.go:356] "Generic (PLEG): container finished" podID="1c9dfbdd-401e-4f49-bb40-176392c82957" containerID="1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d" exitCode=0 Oct 29 23:22:17.075865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.075675 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" Oct 29 23:22:17.075865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.075733 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" event={"ID":"1c9dfbdd-401e-4f49-bb40-176392c82957","Type":"ContainerDied","Data":"1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d"} Oct 29 23:22:17.075865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.075787 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm" event={"ID":"1c9dfbdd-401e-4f49-bb40-176392c82957","Type":"ContainerDied","Data":"d19a97127f77f839000500f973c5224e9650776f4aeb4b60ef63264ccc348ce6"} Oct 29 23:22:17.075865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.075808 5736 scope.go:122] "RemoveContainer" containerID="1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d" Oct 29 23:22:17.088949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.088923 5736 scope.go:122] "RemoveContainer" containerID="1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d" Oct 29 23:22:17.089275 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:22:17.089247 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d\": container with ID starting with 1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d not found: ID does not exist" containerID="1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d" Oct 29 23:22:17.089365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.089288 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d"} err="failed to get container status \"1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d\": rpc error: code = NotFound desc = could not find container \"1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d\": container with ID starting with 1307558bc62cb6aaea74a846c3a03fbad0fc8237efcc70fb836b8f43b2bd288d not found: ID does not exist" Oct 29 23:22:17.189741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.189687 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-dshm\") pod \"1c9dfbdd-401e-4f49-bb40-176392c82957\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " Oct 29 23:22:17.190014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.189991 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-tmp-dir\") pod \"1c9dfbdd-401e-4f49-bb40-176392c82957\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " Oct 29 23:22:17.190191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.190177 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-home\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-home\") pod \"1c9dfbdd-401e-4f49-bb40-176392c82957\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " Oct 29 23:22:17.190347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.190322 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-tmp-dir" pod "1c9dfbdd-401e-4f49-bb40-176392c82957" (UID: "1c9dfbdd-401e-4f49-bb40-176392c82957"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:17.190473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.190455 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9dfbdd-401e-4f49-bb40-176392c82957-kube-api-access-njkqn\" (UniqueName: \"kubernetes.io/projected/1c9dfbdd-401e-4f49-bb40-176392c82957-kube-api-access-njkqn\") pod \"1c9dfbdd-401e-4f49-bb40-176392c82957\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " Oct 29 23:22:17.190632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.190614 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c9dfbdd-401e-4f49-bb40-176392c82957-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9dfbdd-401e-4f49-bb40-176392c82957-tls-certs\") pod \"1c9dfbdd-401e-4f49-bb40-176392c82957\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " Oct 29 23:22:17.190756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.190732 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-home" pod "1c9dfbdd-401e-4f49-bb40-176392c82957" (UID: "1c9dfbdd-401e-4f49-bb40-176392c82957"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:17.190932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.190905 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-model-cache\") pod \"1c9dfbdd-401e-4f49-bb40-176392c82957\" (UID: \"1c9dfbdd-401e-4f49-bb40-176392c82957\") " Oct 29 23:22:17.192370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.192328 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.192370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.192355 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.194856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.194824 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-model-cache" pod "1c9dfbdd-401e-4f49-bb40-176392c82957" (UID: "1c9dfbdd-401e-4f49-bb40-176392c82957"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:17.195211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.195164 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" podUID="87e5da3a-f63d-4133-a5cb-90bc11aaa123" containerName="main" containerID="cri-o://de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed" gracePeriod=45 Oct 29 23:22:17.197283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.197243 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9dfbdd-401e-4f49-bb40-176392c82957-kube-api-access-njkqn" pod "1c9dfbdd-401e-4f49-bb40-176392c82957" (UID: "1c9dfbdd-401e-4f49-bb40-176392c82957"). InnerVolumeSpecName "kube-api-access-njkqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:17.197615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.197571 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9dfbdd-401e-4f49-bb40-176392c82957-tls-certs" pod "1c9dfbdd-401e-4f49-bb40-176392c82957" (UID: "1c9dfbdd-401e-4f49-bb40-176392c82957"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:22:17.209796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.209772 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-dshm" pod "1c9dfbdd-401e-4f49-bb40-176392c82957" (UID: "1c9dfbdd-401e-4f49-bb40-176392c82957"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:17.237734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.237696 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5"] Oct 29 23:22:17.294798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.294727 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njkqn\" (UniqueName: \"kubernetes.io/projected/1c9dfbdd-401e-4f49-bb40-176392c82957-kube-api-access-njkqn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.294798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.294763 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9dfbdd-401e-4f49-bb40-176392c82957-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.294798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.294779 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.294798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.294791 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9dfbdd-401e-4f49-bb40-176392c82957-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.501523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.501475 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" podUID="87e5da3a-f63d-4133-a5cb-90bc11aaa123" containerName="main" probeResult="failure" output="Get \"https://10.134.0.239:8000/ready\": dial tcp 10.134.0.239:8000: connect: connection refused" Oct 29 23:22:17.511494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.511443 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm"] Oct 29 23:22:17.521647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.521616 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-26a2a4/auth-test-model-26a2a4-kserve-7fd6674ff7-zlbmm"] Oct 29 23:22:17.624387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.624363 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:22:17.779629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.779562 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" podUID="4eec5768-62f2-4d97-b6dc-4f9971e8e919" containerName="main" containerID="cri-o://99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2" gracePeriod=45 Oct 29 23:22:17.805492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.805466 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tls-certs\" (UniqueName: \"kubernetes.io/secret/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tls-certs\") pod \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " Oct 29 23:22:17.805634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.805542 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tmp-dir\") pod \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " Oct 29 23:22:17.805702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.805640 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-model-cache\") pod \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " Oct 29 23:22:17.805769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.805725 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-home\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-home\") pod \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " Oct 29 23:22:17.805830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.805781 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-dshm\") pod \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " Oct 29 23:22:17.805883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.805835 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tmp-dir" pod "87e5da3a-f63d-4133-a5cb-90bc11aaa123" (UID: "87e5da3a-f63d-4133-a5cb-90bc11aaa123"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:17.805941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.805899 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e5da3a-f63d-4133-a5cb-90bc11aaa123-kube-api-access-78sxt\" (UniqueName: \"kubernetes.io/projected/87e5da3a-f63d-4133-a5cb-90bc11aaa123-kube-api-access-78sxt\") pod \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\" (UID: \"87e5da3a-f63d-4133-a5cb-90bc11aaa123\") " Oct 29 23:22:17.805941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.805912 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-model-cache" pod "87e5da3a-f63d-4133-a5cb-90bc11aaa123" (UID: "87e5da3a-f63d-4133-a5cb-90bc11aaa123"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:17.806051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.805952 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-home" pod "87e5da3a-f63d-4133-a5cb-90bc11aaa123" (UID: "87e5da3a-f63d-4133-a5cb-90bc11aaa123"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:17.806612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.806559 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.806612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.806595 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.806612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.806608 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.807888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.807863 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-dshm" pod "87e5da3a-f63d-4133-a5cb-90bc11aaa123" (UID: "87e5da3a-f63d-4133-a5cb-90bc11aaa123"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:17.808610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.808573 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tls-certs" pod "87e5da3a-f63d-4133-a5cb-90bc11aaa123" (UID: "87e5da3a-f63d-4133-a5cb-90bc11aaa123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:22:17.808701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.808633 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e5da3a-f63d-4133-a5cb-90bc11aaa123-kube-api-access-78sxt" pod "87e5da3a-f63d-4133-a5cb-90bc11aaa123" (UID: "87e5da3a-f63d-4133-a5cb-90bc11aaa123"). InnerVolumeSpecName "kube-api-access-78sxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:17.907882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.907855 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e5da3a-f63d-4133-a5cb-90bc11aaa123-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.907882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.907882 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e5da3a-f63d-4133-a5cb-90bc11aaa123-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:17.908035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:17.907893 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78sxt\" (UniqueName: \"kubernetes.io/projected/87e5da3a-f63d-4133-a5cb-90bc11aaa123-kube-api-access-78sxt\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:18.063264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.063233 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:22:18.084704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.084672 5736 generic.go:356] "Generic (PLEG): container finished" podID="87e5da3a-f63d-4133-a5cb-90bc11aaa123" containerID="de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed" exitCode=0 Oct 29 23:22:18.084853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.084717 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" Oct 29 23:22:18.084853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.084740 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" event={"ID":"87e5da3a-f63d-4133-a5cb-90bc11aaa123","Type":"ContainerDied","Data":"de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed"} Oct 29 23:22:18.084853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.084786 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj" event={"ID":"87e5da3a-f63d-4133-a5cb-90bc11aaa123","Type":"ContainerDied","Data":"a5b9ca71581681e5b50ac36bfa48b717b92d5105f1738f9dde2a4ac894d86194"} Oct 29 23:22:18.084853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.084804 5736 scope.go:122] "RemoveContainer" containerID="de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed" Oct 29 23:22:18.086422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.086400 5736 generic.go:356] "Generic (PLEG): container finished" podID="4eec5768-62f2-4d97-b6dc-4f9971e8e919" containerID="99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2" exitCode=0 Oct 29 23:22:18.086547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.086445 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" Oct 29 23:22:18.086547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.086479 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" event={"ID":"4eec5768-62f2-4d97-b6dc-4f9971e8e919","Type":"ContainerDied","Data":"99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2"} Oct 29 23:22:18.086547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.086506 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm" event={"ID":"4eec5768-62f2-4d97-b6dc-4f9971e8e919","Type":"ContainerDied","Data":"c3e06fb324fee083c27eccff0ce2eff4d8bdf172906c9097812b6e568a493657"} Oct 29 23:22:18.096022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.096006 5736 scope.go:122] "RemoveContainer" containerID="de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed" Oct 29 23:22:18.096305 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:22:18.096289 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed\": container with ID starting with de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed not found: ID does not exist" containerID="de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed" Oct 29 23:22:18.096356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.096315 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed"} err="failed to get container status \"de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed\": rpc error: code = NotFound desc = could not find container \"de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed\": container with ID starting with de7660018e55a9bcf6197bf2e6cb13f18aade5eabe8fc04a7b5ba7451a4718ed not found: ID does not exist" Oct 29 23:22:18.096356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.096330 5736 scope.go:122] "RemoveContainer" containerID="99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2" Oct 29 23:22:18.107002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.106983 5736 scope.go:122] "RemoveContainer" containerID="99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2" Oct 29 23:22:18.107300 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:22:18.107282 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2\": container with ID starting with 99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2 not found: ID does not exist" containerID="99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2" Oct 29 23:22:18.107368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.107310 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2"} err="failed to get container status \"99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2\": rpc error: code = NotFound desc = could not find container \"99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2\": container with ID starting with 99fe5ba2189141caec99b17982957260746522381fad13d2456e44d8ce786cf2 not found: ID does not exist" Oct 29 23:22:18.168787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.168715 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj"] Oct 29 23:22:18.182761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.182733 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2126fb/auth-test-model-2126fb-kserve-84ccbb44bf-5bnbj"] Oct 29 23:22:18.212294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.212271 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eec5768-62f2-4d97-b6dc-4f9971e8e919-kube-api-access-vg4wv\" (UniqueName: \"kubernetes.io/projected/4eec5768-62f2-4d97-b6dc-4f9971e8e919-kube-api-access-vg4wv\") pod \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " Oct 29 23:22:18.212395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.212330 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-dshm\") pod \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " Oct 29 23:22:18.212395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.212383 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tmp-dir\") pod \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " Oct 29 23:22:18.212512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.212440 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-home\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-home\") pod \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " Oct 29 23:22:18.212512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.212469 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-model-cache\") pod \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " Oct 29 23:22:18.212639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.212567 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tls-certs\") pod \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\" (UID: \"4eec5768-62f2-4d97-b6dc-4f9971e8e919\") " Oct 29 23:22:18.212783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.212751 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-home" pod "4eec5768-62f2-4d97-b6dc-4f9971e8e919" (UID: "4eec5768-62f2-4d97-b6dc-4f9971e8e919"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:18.212783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.212768 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tmp-dir" pod "4eec5768-62f2-4d97-b6dc-4f9971e8e919" (UID: "4eec5768-62f2-4d97-b6dc-4f9971e8e919"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:18.212962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.212853 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-model-cache" pod "4eec5768-62f2-4d97-b6dc-4f9971e8e919" (UID: "4eec5768-62f2-4d97-b6dc-4f9971e8e919"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:18.213351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.213330 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:18.213422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.213353 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:18.213422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.213366 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:18.214616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.214595 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-dshm" pod "4eec5768-62f2-4d97-b6dc-4f9971e8e919" (UID: "4eec5768-62f2-4d97-b6dc-4f9971e8e919"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:18.214697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.214679 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eec5768-62f2-4d97-b6dc-4f9971e8e919-kube-api-access-vg4wv" pod "4eec5768-62f2-4d97-b6dc-4f9971e8e919" (UID: "4eec5768-62f2-4d97-b6dc-4f9971e8e919"). InnerVolumeSpecName "kube-api-access-vg4wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:18.214738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.214719 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tls-certs" pod "4eec5768-62f2-4d97-b6dc-4f9971e8e919" (UID: "4eec5768-62f2-4d97-b6dc-4f9971e8e919"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:22:18.316603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.315363 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4wv\" (UniqueName: \"kubernetes.io/projected/4eec5768-62f2-4d97-b6dc-4f9971e8e919-kube-api-access-vg4wv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:18.316603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.315404 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eec5768-62f2-4d97-b6dc-4f9971e8e919-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:18.316603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.315419 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eec5768-62f2-4d97-b6dc-4f9971e8e919-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:18.433228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.433150 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm"] Oct 29 23:22:18.437134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:18.437112 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b73a75/auth-test-model-b73a75-kserve-7bb9c47f8f-9mcjm"] Oct 29 23:22:19.005837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:19.005767 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9dfbdd-401e-4f49-bb40-176392c82957" path="/var/lib/kubelet/pods/1c9dfbdd-401e-4f49-bb40-176392c82957/volumes" Oct 29 23:22:19.006319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:19.006303 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eec5768-62f2-4d97-b6dc-4f9971e8e919" path="/var/lib/kubelet/pods/4eec5768-62f2-4d97-b6dc-4f9971e8e919/volumes" Oct 29 23:22:19.006801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:19.006787 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e5da3a-f63d-4133-a5cb-90bc11aaa123" path="/var/lib/kubelet/pods/87e5da3a-f63d-4133-a5cb-90bc11aaa123/volumes" Oct 29 23:22:22.972013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:22.971977 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g"] Oct 29 23:22:23.115373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.115322 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" podUID="a216c693-7350-49a8-b979-a706ae70da07" containerName="main" containerID="cri-o://fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c" gracePeriod=45 Oct 29 23:22:23.618595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.618553 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:22:23.645288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.645256 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-model-cache\") pod \"a216c693-7350-49a8-b979-a706ae70da07\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " Oct 29 23:22:23.645420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.645382 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-dshm\") pod \"a216c693-7350-49a8-b979-a706ae70da07\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " Oct 29 23:22:23.645420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.645407 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a216c693-7350-49a8-b979-a706ae70da07-kube-api-access-bx8r5\" (UniqueName: \"kubernetes.io/projected/a216c693-7350-49a8-b979-a706ae70da07-kube-api-access-bx8r5\") pod \"a216c693-7350-49a8-b979-a706ae70da07\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " Oct 29 23:22:23.645508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.645444 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-tmp-dir\") pod \"a216c693-7350-49a8-b979-a706ae70da07\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " Oct 29 23:22:23.645552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.645509 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-model-cache" pod "a216c693-7350-49a8-b979-a706ae70da07" (UID: "a216c693-7350-49a8-b979-a706ae70da07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:23.645633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.645548 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-home\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-home\") pod \"a216c693-7350-49a8-b979-a706ae70da07\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " Oct 29 23:22:23.645699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.645659 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a216c693-7350-49a8-b979-a706ae70da07-tls-certs\" (UniqueName: \"kubernetes.io/secret/a216c693-7350-49a8-b979-a706ae70da07-tls-certs\") pod \"a216c693-7350-49a8-b979-a706ae70da07\" (UID: \"a216c693-7350-49a8-b979-a706ae70da07\") " Oct 29 23:22:23.645776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.645753 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-tmp-dir" pod "a216c693-7350-49a8-b979-a706ae70da07" (UID: "a216c693-7350-49a8-b979-a706ae70da07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:23.645891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.645870 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-home" pod "a216c693-7350-49a8-b979-a706ae70da07" (UID: "a216c693-7350-49a8-b979-a706ae70da07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:23.646147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.646122 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:23.646147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.646149 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:23.646350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.646160 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:23.647709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.647687 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-dshm" pod "a216c693-7350-49a8-b979-a706ae70da07" (UID: "a216c693-7350-49a8-b979-a706ae70da07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:23.647935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.647903 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a216c693-7350-49a8-b979-a706ae70da07-tls-certs" pod "a216c693-7350-49a8-b979-a706ae70da07" (UID: "a216c693-7350-49a8-b979-a706ae70da07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:22:23.648080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.647946 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a216c693-7350-49a8-b979-a706ae70da07-kube-api-access-bx8r5" pod "a216c693-7350-49a8-b979-a706ae70da07" (UID: "a216c693-7350-49a8-b979-a706ae70da07"). InnerVolumeSpecName "kube-api-access-bx8r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:23.750435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.747408 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a216c693-7350-49a8-b979-a706ae70da07-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:23.750435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.747444 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8r5\" (UniqueName: \"kubernetes.io/projected/a216c693-7350-49a8-b979-a706ae70da07-kube-api-access-bx8r5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:23.750435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:23.747471 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a216c693-7350-49a8-b979-a706ae70da07-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:24.154018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:24.153987 5736 generic.go:356] "Generic (PLEG): container finished" podID="a216c693-7350-49a8-b979-a706ae70da07" containerID="fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c" exitCode=0 Oct 29 23:22:24.154431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:24.154042 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" Oct 29 23:22:24.154431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:24.154074 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" event={"ID":"a216c693-7350-49a8-b979-a706ae70da07","Type":"ContainerDied","Data":"fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c"} Oct 29 23:22:24.154431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:24.154121 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c" event={"ID":"a216c693-7350-49a8-b979-a706ae70da07","Type":"ContainerDied","Data":"0ebebf9413ad7da76263b46fb9fd85eb6a78441790c68ece76ab2cf9aa3c1775"} Oct 29 23:22:24.154431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:24.154144 5736 scope.go:122] "RemoveContainer" containerID="fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c" Oct 29 23:22:24.166528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:24.166510 5736 scope.go:122] "RemoveContainer" containerID="fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c" Oct 29 23:22:24.166866 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:22:24.166839 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c\": container with ID starting with fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c not found: ID does not exist" containerID="fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c" Oct 29 23:22:24.166975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:24.166874 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c"} err="failed to get container status \"fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c\": rpc error: code = NotFound desc = could not find container \"fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c\": container with ID starting with fa25c4b23448263121d5a48cf218093e9d942b61d0c53111a0d4bfae2253546c not found: ID does not exist" Oct 29 23:22:24.203977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:24.203951 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c"] Oct 29 23:22:24.213498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:24.213476 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b4b5e3/auth-test-model-b4b5e3-kserve-5d5c656885-pbt9c"] Oct 29 23:22:25.012796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.012758 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a216c693-7350-49a8-b979-a706ae70da07" path="/var/lib/kubelet/pods/a216c693-7350-49a8-b979-a706ae70da07/volumes" Oct 29 23:22:25.013617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.013573 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4"] Oct 29 23:22:25.014596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.014558 5736 cpu_manager.go:414] "containerMap: removing container" podUID="87e5da3a-f63d-4133-a5cb-90bc11aaa123" containerName="main" Oct 29 23:22:25.014827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.014806 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e5da3a-f63d-4133-a5cb-90bc11aaa123" containerName="main" Oct 29 23:22:25.014903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.014845 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a216c693-7350-49a8-b979-a706ae70da07" containerName="main" Oct 29 23:22:25.014903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.014853 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a216c693-7350-49a8-b979-a706ae70da07" containerName="main" Oct 29 23:22:25.014903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.014872 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1c9dfbdd-401e-4f49-bb40-176392c82957" containerName="main" Oct 29 23:22:25.014903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.014879 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9dfbdd-401e-4f49-bb40-176392c82957" containerName="main" Oct 29 23:22:25.014903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.014899 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4eec5768-62f2-4d97-b6dc-4f9971e8e919" containerName="main" Oct 29 23:22:25.015140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.014907 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eec5768-62f2-4d97-b6dc-4f9971e8e919" containerName="main" Oct 29 23:22:25.015140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.015090 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e5da3a-f63d-4133-a5cb-90bc11aaa123" containerName="main" Oct 29 23:22:25.015140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.015109 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a216c693-7350-49a8-b979-a706ae70da07" containerName="main" Oct 29 23:22:25.015140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.015120 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9dfbdd-401e-4f49-bb40-176392c82957" containerName="main" Oct 29 23:22:25.015140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.015131 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eec5768-62f2-4d97-b6dc-4f9971e8e919" containerName="main" Oct 29 23:22:25.021063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.021042 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.034453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.034427 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4"] Oct 29 23:22:25.039143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.039122 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c5ef2f-openshift-default-dockercfg-p88ll\"" Oct 29 23:22:25.066698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.066664 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-credential-socket\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.066798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.066758 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-istio-token\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.066887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.066875 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-socket\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.066937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.066903 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04a1403-813a-492d-8472-978da4638bad-istio-podinfo\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.066976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.066949 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-certs\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.067019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.067010 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-envoy\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.067056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.067027 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdcb\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-kube-api-access-9fdcb\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.067056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.067051 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-data\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.067158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.067095 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04a1403-813a-492d-8472-978da4638bad-istiod-ca-cert\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.195069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.195024 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-socket\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.214169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.201633 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-socket\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.215039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.214448 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04a1403-813a-492d-8472-978da4638bad-istio-podinfo\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.215305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.215284 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-certs\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.215848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.215825 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-envoy\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.216371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.216350 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdcb\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-kube-api-access-9fdcb\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.216544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.216527 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-data\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.217052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.217031 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04a1403-813a-492d-8472-978da4638bad-istiod-ca-cert\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.217966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.217945 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-credential-socket\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.218569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.218540 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-istio-token\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.219089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.216931 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-data\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.219229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.218328 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-credential-socket\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.219337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.215703 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-certs\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.219457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.217856 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04a1403-813a-492d-8472-978da4638bad-istiod-ca-cert\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.230611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.225859 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-envoy\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.245607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.245520 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdcb\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-kube-api-access-9fdcb\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.253695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.250424 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04a1403-813a-492d-8472-978da4638bad-istio-podinfo\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.279879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.275559 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-istio-token\") pod \"e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.349380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.349049 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:25.872814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:25.872718 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4"] Oct 29 23:22:26.124680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.118399 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:26.124680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.118498 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:26.124680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.118540 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:26.225320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.225279 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" event={"ID":"b04a1403-813a-492d-8472-978da4638bad","Type":"ContainerStarted","Data":"570e86e0b4e02d4925e0be4773903a41190b2c5ee235681d3e259cb7119d7f42"} Oct 29 23:22:26.334611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.333874 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" podUID="e97569b6-4ebf-4501-9897-a8c34b5c68ef" containerName="main" containerID="cri-o://e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be" gracePeriod=45 Oct 29 23:22:26.473498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.473460 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" podUID="e97569b6-4ebf-4501-9897-a8c34b5c68ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.245:8000/ready\": dial tcp 10.134.0.245:8000: connect: connection refused" Oct 29 23:22:26.634735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.634704 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:22:26.658459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.658416 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-dshm\") pod \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " Oct 29 23:22:26.658618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.658496 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tmp-dir\") pod \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " Oct 29 23:22:26.658714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.658665 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97569b6-4ebf-4501-9897-a8c34b5c68ef-kube-api-access-vvzpz\" (UniqueName: \"kubernetes.io/projected/e97569b6-4ebf-4501-9897-a8c34b5c68ef-kube-api-access-vvzpz\") pod \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " Oct 29 23:22:26.658783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.658720 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tls-certs\") pod \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " Oct 29 23:22:26.658851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.658823 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tmp-dir" pod "e97569b6-4ebf-4501-9897-a8c34b5c68ef" (UID: "e97569b6-4ebf-4501-9897-a8c34b5c68ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:26.658904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.658871 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-home\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-home\") pod \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " Oct 29 23:22:26.658961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.658913 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-model-cache\") pod \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\" (UID: \"e97569b6-4ebf-4501-9897-a8c34b5c68ef\") " Oct 29 23:22:26.659112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.659091 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-home" pod "e97569b6-4ebf-4501-9897-a8c34b5c68ef" (UID: "e97569b6-4ebf-4501-9897-a8c34b5c68ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:26.659394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.659373 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-model-cache" pod "e97569b6-4ebf-4501-9897-a8c34b5c68ef" (UID: "e97569b6-4ebf-4501-9897-a8c34b5c68ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:26.659797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.659771 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:26.660047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.659800 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:26.660047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.659819 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:26.661340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.661313 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-dshm" pod "e97569b6-4ebf-4501-9897-a8c34b5c68ef" (UID: "e97569b6-4ebf-4501-9897-a8c34b5c68ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:26.661449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.661368 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97569b6-4ebf-4501-9897-a8c34b5c68ef-kube-api-access-vvzpz" pod "e97569b6-4ebf-4501-9897-a8c34b5c68ef" (UID: "e97569b6-4ebf-4501-9897-a8c34b5c68ef"). InnerVolumeSpecName "kube-api-access-vvzpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:26.661665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.661643 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tls-certs" pod "e97569b6-4ebf-4501-9897-a8c34b5c68ef" (UID: "e97569b6-4ebf-4501-9897-a8c34b5c68ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:22:26.701736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.701664 5736 scope.go:122] "RemoveContainer" containerID="8206797de78d7b2eaa7eae8a4b340fa1114c0fd8e7c287a73d24f9aefb6a1518" Oct 29 23:22:26.760814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.760789 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97569b6-4ebf-4501-9897-a8c34b5c68ef-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:26.760814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.760815 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvzpz\" (UniqueName: \"kubernetes.io/projected/e97569b6-4ebf-4501-9897-a8c34b5c68ef-kube-api-access-vvzpz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:26.760944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:26.760825 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97569b6-4ebf-4501-9897-a8c34b5c68ef-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:26.825825 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:22:26.825790 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:22:27.238764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.238725 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" event={"ID":"b04a1403-813a-492d-8472-978da4638bad","Type":"ContainerStarted","Data":"a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01"} Oct 29 23:22:27.240384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.240357 5736 generic.go:356] "Generic (PLEG): container finished" podID="e97569b6-4ebf-4501-9897-a8c34b5c68ef" containerID="e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be" exitCode=0 Oct 29 23:22:27.240542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.240397 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" Oct 29 23:22:27.240542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.240397 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" event={"ID":"e97569b6-4ebf-4501-9897-a8c34b5c68ef","Type":"ContainerDied","Data":"e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be"} Oct 29 23:22:27.240542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.240443 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm" event={"ID":"e97569b6-4ebf-4501-9897-a8c34b5c68ef","Type":"ContainerDied","Data":"aaef4bc65cce813f5708f1622dfd3157c359828f0a8452f859c3bbd40541f4f9"} Oct 29 23:22:27.240542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.240468 5736 scope.go:122] "RemoveContainer" containerID="e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be" Oct 29 23:22:27.251199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.251177 5736 scope.go:122] "RemoveContainer" containerID="e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be" Oct 29 23:22:27.251481 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:22:27.251462 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be\": container with ID starting with e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be not found: ID does not exist" containerID="e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be" Oct 29 23:22:27.251540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.251491 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be"} err="failed to get container status \"e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be\": rpc error: code = NotFound desc = could not find container \"e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be\": container with ID starting with e413df84c54af7666f4ba4f83229090aa672ccc7fe4d6553b71b1b0a950323be not found: ID does not exist" Oct 29 23:22:27.302136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.302055 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" podStartSLOduration=3.302043704 podStartE2EDuration="3.302043704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:27.292779178 +0000 UTC m=+2344.408698783" watchObservedRunningTime="2026-10-29 23:22:27.302043704 +0000 UTC m=+2344.417963310" Oct 29 23:22:27.332195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.332165 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm"] Oct 29 23:22:27.339571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.339545 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8a8b61/auth-test-model-8a8b61-kserve-5d79c5cccc-hvxfm"] Oct 29 23:22:27.349904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.349869 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:27.356772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:27.356750 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:28.306654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:28.298125 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:28.309832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:28.308846 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:22:29.005395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:29.005361 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97569b6-4ebf-4501-9897-a8c34b5c68ef" path="/var/lib/kubelet/pods/e97569b6-4ebf-4501-9897-a8c34b5c68ef/volumes" Oct 29 23:22:32.280197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.276544 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" podUID="b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" containerName="main" containerID="cri-o://1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9" gracePeriod=45 Oct 29 23:22:32.329663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.326824 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" podUID="b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.243:8000/ready\": read tcp 10.134.0.2:38880->10.134.0.243:8000: read: connection reset by peer" Oct 29 23:22:32.604187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.604159 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:22:32.668575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.668542 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-model-cache\") pod \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " Oct 29 23:22:32.668774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.668666 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-kube-api-access-mjn8b\" (UniqueName: \"kubernetes.io/projected/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-kube-api-access-mjn8b\") pod \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " Oct 29 23:22:32.668774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.668719 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-home\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-home\") pod \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " Oct 29 23:22:32.668774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.668759 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-dshm\") pod \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " Oct 29 23:22:32.668965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.668900 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tls-certs\") pod \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " Oct 29 23:22:32.669023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.668994 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tmp-dir\") pod \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\" (UID: \"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9\") " Oct 29 23:22:32.670146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.669696 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-home" pod "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" (UID: "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:32.670146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.669704 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tmp-dir" pod "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" (UID: "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:32.670146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.669719 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-model-cache" pod "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" (UID: "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:32.671878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.671852 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tls-certs" pod "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" (UID: "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:22:32.671982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.671891 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-kube-api-access-mjn8b" pod "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" (UID: "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9"). InnerVolumeSpecName "kube-api-access-mjn8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:32.672475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.672452 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-dshm" pod "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" (UID: "b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:32.771092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.771061 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:32.771092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.771090 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:32.771092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.771099 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjn8b\" (UniqueName: \"kubernetes.io/projected/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-kube-api-access-mjn8b\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:32.771318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.771108 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:32.771318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.771118 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:32.771318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:32.771125 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:33.377153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:33.377078 5736 generic.go:356] "Generic (PLEG): container finished" podID="b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" containerID="1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9" exitCode=0 Oct 29 23:22:33.377153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:33.377121 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" Oct 29 23:22:33.377649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:33.377166 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" event={"ID":"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9","Type":"ContainerDied","Data":"1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9"} Oct 29 23:22:33.377649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:33.377214 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5" event={"ID":"b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9","Type":"ContainerDied","Data":"9b1d6cffa8a587d3814b28f1ee7edd2953e441ae7387753df65d358afaf69e10"} Oct 29 23:22:33.377649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:33.377235 5736 scope.go:122] "RemoveContainer" containerID="1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9" Oct 29 23:22:33.387703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:33.387680 5736 scope.go:122] "RemoveContainer" containerID="1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9" Oct 29 23:22:33.387978 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:22:33.387957 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9\": container with ID starting with 1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9 not found: ID does not exist" containerID="1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9" Oct 29 23:22:33.388056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:33.387991 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9"} err="failed to get container status \"1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9\": rpc error: code = NotFound desc = could not find container \"1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9\": container with ID starting with 1c246600bc90eda68caccc703582f470885a1bcafee3588729372d57cb535af9 not found: ID does not exist" Oct 29 23:22:33.442827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:33.441562 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5"] Oct 29 23:22:33.449564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:33.449528 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-52f67b/auth-test-model-52f67b-kserve-6475fb5cf7-f4lx5"] Oct 29 23:22:35.004902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:35.004805 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" path="/var/lib/kubelet/pods/b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9/volumes" Oct 29 23:22:37.993557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:37.993502 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" podUID="92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" containerName="main" containerID="cri-o://d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0" gracePeriod=45 Oct 29 23:22:38.282285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.282254 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:22:38.423785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.423748 5736 generic.go:356] "Generic (PLEG): container finished" podID="92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" containerID="d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0" exitCode=0 Oct 29 23:22:38.423985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.423793 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" Oct 29 23:22:38.423985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.423829 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" event={"ID":"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7","Type":"ContainerDied","Data":"d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0"} Oct 29 23:22:38.423985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.423884 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g" event={"ID":"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7","Type":"ContainerDied","Data":"18c358e55ba3306cc7869e271ec52c41f8bda9b1290ae7690654569eff009865"} Oct 29 23:22:38.423985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.423907 5736 scope.go:122] "RemoveContainer" containerID="d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0" Oct 29 23:22:38.461715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.461679 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-dshm\") pod \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " Oct 29 23:22:38.461901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.461881 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tls-certs\") pod \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " Oct 29 23:22:38.469770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.469729 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-home\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-home\") pod \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " Oct 29 23:22:38.470104 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.470077 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-model-cache\") pod \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " Oct 29 23:22:38.470541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.470523 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tmp-dir\") pod \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " Oct 29 23:22:38.471098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.471078 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-kube-api-access-872zs\" (UniqueName: \"kubernetes.io/projected/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-kube-api-access-872zs\") pod \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\" (UID: \"92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7\") " Oct 29 23:22:38.473660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.470444 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-model-cache" pod "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" (UID: "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:38.473811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.470904 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tmp-dir" pod "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" (UID: "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:38.474021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.472559 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-home" pod "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" (UID: "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:38.474446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.474384 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:38.474446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.474403 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:38.474446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.474430 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:38.478495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.478469 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-dshm" pod "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" (UID: "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:38.479770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.479744 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-kube-api-access-872zs" pod "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" (UID: "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7"). InnerVolumeSpecName "kube-api-access-872zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:38.480185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.480152 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tls-certs" pod "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" (UID: "92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:22:38.514552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.514522 5736 scope.go:122] "RemoveContainer" containerID="d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0" Oct 29 23:22:38.514890 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:22:38.514867 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0\": container with ID starting with d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0 not found: ID does not exist" containerID="d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0" Oct 29 23:22:38.514959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.514900 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0"} err="failed to get container status \"d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0\": rpc error: code = NotFound desc = could not find container \"d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0\": container with ID starting with d4fef4f3d1c0113785f0808b27aae588f0c14ca5e1610bb8c6f6f9324de800c0 not found: ID does not exist" Oct 29 23:22:38.576331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.576295 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:38.576331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.576327 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872zs\" (UniqueName: \"kubernetes.io/projected/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-kube-api-access-872zs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:38.576331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.576337 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:22:38.766117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.766030 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g"] Oct 29 23:22:38.775691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:38.775663 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c83b50/auth-test-model-c83b50-kserve-66f4f969f7-ftg4g"] Oct 29 23:22:39.005010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:39.004970 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" path="/var/lib/kubelet/pods/92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7/volumes" Oct 29 23:22:42.633275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.633238 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf"] Oct 29 23:22:42.633963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.633941 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" containerName="main" Oct 29 23:22:42.633963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.633964 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" containerName="main" Oct 29 23:22:42.634153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.633986 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e97569b6-4ebf-4501-9897-a8c34b5c68ef" containerName="main" Oct 29 23:22:42.634153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.633995 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97569b6-4ebf-4501-9897-a8c34b5c68ef" containerName="main" Oct 29 23:22:42.634153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.634022 5736 cpu_manager.go:414] "containerMap: removing container" podUID="92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" containerName="main" Oct 29 23:22:42.634153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.634031 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" containerName="main" Oct 29 23:22:42.634346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.634187 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ff4b26-e8fe-4f49-9b5b-01df9b2e16c7" containerName="main" Oct 29 23:22:42.634346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.634207 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a3bd94-e937-433e-9d7c-6ae6fe8f4be9" containerName="main" Oct 29 23:22:42.634346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.634214 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97569b6-4ebf-4501-9897-a8c34b5c68ef" containerName="main" Oct 29 23:22:42.639282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.639260 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.652838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.652817 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-8a8b61-openshift-default-dockercfg-lrtvw\"" Oct 29 23:22:42.667046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.667021 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf"] Oct 29 23:22:42.748142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.748108 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-credential-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.748331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.748148 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbq7\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-kube-api-access-6wbq7\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.748331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.748235 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-envoy\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.748463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.748396 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-data\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.748463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.748434 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.748545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.748470 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7924e4-0728-45ba-9f05-1a4e1d725062-istiod-ca-cert\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.748545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.748527 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-certs\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.748713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.748689 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-token\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.748829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.748731 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-podinfo\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.849874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.849838 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-token\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.849874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.849876 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-podinfo\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.849996 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-credential-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850058 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbq7\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-kube-api-access-6wbq7\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850132 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-envoy\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850268 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-data\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850300 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850324 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7924e4-0728-45ba-9f05-1a4e1d725062-istiod-ca-cert\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850359 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-certs\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850374 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-credential-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850620 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-data\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850684 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-socket\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.850727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.850703 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-certs\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.851040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.851019 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7924e4-0728-45ba-9f05-1a4e1d725062-istiod-ca-cert\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.852885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.852861 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-envoy\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.853224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.853204 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-podinfo\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.871276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.871244 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-token\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.871375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.871307 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbq7\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-kube-api-access-6wbq7\") pod \"e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:42.953503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:42.953442 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:43.284555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:43.284529 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf"] Oct 29 23:22:43.286877 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:22:43.286845 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7924e4_0728_45ba_9f05_1a4e1d725062.slice/crio-b68f6ce4618544363fc3e78a2c97635669a329e514d411074169a62ca47c35d7 WatchSource:0}: Error finding container b68f6ce4618544363fc3e78a2c97635669a329e514d411074169a62ca47c35d7: Status 404 returned error can't find the container with id b68f6ce4618544363fc3e78a2c97635669a329e514d411074169a62ca47c35d7 Oct 29 23:22:43.289748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:43.289717 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:43.289833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:43.289780 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:43.289833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:43.289808 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:43.466933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:43.466878 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" event={"ID":"4a7924e4-0728-45ba-9f05-1a4e1d725062","Type":"ContainerStarted","Data":"82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533"} Oct 29 23:22:43.467112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:43.466955 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" event={"ID":"4a7924e4-0728-45ba-9f05-1a4e1d725062","Type":"ContainerStarted","Data":"b68f6ce4618544363fc3e78a2c97635669a329e514d411074169a62ca47c35d7"} Oct 29 23:22:43.527105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:43.527044 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" podStartSLOduration=1.527030383 podStartE2EDuration="1.527030383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:43.521804623 +0000 UTC m=+2360.637724231" watchObservedRunningTime="2026-10-29 23:22:43.527030383 +0000 UTC m=+2360.642949989" Oct 29 23:22:43.953858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:43.953824 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:44.957310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:44.957257 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 29 23:22:44.957310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:44.957319 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 29 23:22:45.955977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:45.955938 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 29 23:22:45.956173 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:45.955997 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 29 23:22:46.956287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:46.956247 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 29 23:22:46.956846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:46.956313 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 29 23:22:47.955978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:47.955939 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 29 23:22:47.956296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:47.955998 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 29 23:22:48.955606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:48.954052 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 29 23:22:48.955606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:48.954123 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 29 23:22:49.955711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:49.955669 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 29 23:22:49.956236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:49.955724 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 29 23:22:50.960248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:50.960218 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:50.960770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:50.960610 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:50.961255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:50.961234 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:22:56.826256 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:22:56.826214 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:22:56.888553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:56.888518 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz"] Oct 29 23:22:56.892948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:56.892923 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:56.929629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:56.929572 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz"] Oct 29 23:22:56.982606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:56.980897 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b73a75-openshift-default-dockercfg-vb8t8\"" Oct 29 23:22:57.063540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.063497 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwfb\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-kube-api-access-xxwfb\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.070203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.070164 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-token\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.070452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.070431 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-data\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.070625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.070605 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-certs\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.070761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.070747 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.070930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.070901 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-credential-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.071185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.071164 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-envoy\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.071395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.071379 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-podinfo\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.071752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.071669 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.173839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.173795 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-credential-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.174044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.173996 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-envoy\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.174164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.174141 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-podinfo\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.174315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.174261 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.174399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.174344 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwfb\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-kube-api-access-xxwfb\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.174399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.174374 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-token\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.174522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.174432 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-data\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.174522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.174497 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-certs\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.174522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.174521 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.175647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.175042 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.175647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.175101 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-credential-socket\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.175647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.175301 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-data\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.175647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.175409 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.176173 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.176148 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-certs\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.177962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.177939 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-envoy\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.178321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.178303 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-podinfo\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.195773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.195733 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-token\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.196336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.196312 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwfb\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-kube-api-access-xxwfb\") pod \"e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.209070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.209048 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:57.834602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.833426 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz"] Oct 29 23:22:57.890751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.890707 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:57.890874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.890805 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:57.890874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:57.890854 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:22:58.690325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.690274 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" event={"ID":"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1","Type":"ContainerStarted","Data":"094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15"} Oct 29 23:22:58.690529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.690341 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" event={"ID":"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1","Type":"ContainerStarted","Data":"a9dcee5aa70d6579f32dea4befba6f92f7d802957edf66b99844e189f8868786"} Oct 29 23:22:58.706569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.699773 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5"] Oct 29 23:22:58.706569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.706444 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.721824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.719176 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"payload-processing-external-model-e2e--dc95ba64-dockercfg-wxvth\"" Oct 29 23:22:58.721824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.719344 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"agc-cred-f6e7441f37e1\"" Oct 29 23:22:58.721824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.719890 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"payload-processing-external-model-plugins-e2e-extmod-b-206095\"" Oct 29 23:22:58.730480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.730325 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"routing-overlay\"" Oct 29 23:22:58.744640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.730381 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"openshift-service-ca.crt\"" Oct 29 23:22:58.744804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.732381 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"kube-root-ca.crt\"" Oct 29 23:22:58.744900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.739992 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5"] Oct 29 23:22:58.836642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.817568 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.855062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.855021 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6kg\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-kube-api-access-xl6kg\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.855546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.855524 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.856376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.856350 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.872696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.872283 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" podStartSLOduration=2.872262569 podStartE2EDuration="2.872262569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:58.797070323 +0000 UTC m=+2375.912989942" watchObservedRunningTime="2026-10-29 23:22:58.872262569 +0000 UTC m=+2375.988182177" Oct 29 23:22:58.974086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.974036 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6kg\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-kube-api-access-xl6kg\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.974792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.974766 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.976004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.975978 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.976213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.976195 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.977214 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.977190 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.977360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.975820 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:58.981821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:58.981773 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:59.016066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:59.015996 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6kg\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-kube-api-access-xl6kg\") pod \"payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:59.042360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:59.042002 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:22:59.243661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:59.243327 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:22:59.264729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:59.243984 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 29 23:22:59.264729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:59.244043 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 29 23:22:59.895377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:22:59.895335 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5"] Oct 29 23:22:59.897261 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:22:59.897222 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55be6c9f_0e46_47ce_84a0_0c2dab8bd577.slice/crio-3b17069cc55ed34fd2c03c9c126ef7a15600927c9be76268a5f3bc88a6f3014f WatchSource:0}: Error finding container 3b17069cc55ed34fd2c03c9c126ef7a15600927c9be76268a5f3bc88a6f3014f: Status 404 returned error can't find the container with id 3b17069cc55ed34fd2c03c9c126ef7a15600927c9be76268a5f3bc88a6f3014f Oct 29 23:23:00.209860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:00.209826 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 29 23:23:00.210029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:00.209893 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 29 23:23:00.727896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:00.727846 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" event={"ID":"55be6c9f-0e46-47ce-84a0-0c2dab8bd577","Type":"ContainerStarted","Data":"13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351"} Oct 29 23:23:00.728096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:00.727914 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" event={"ID":"55be6c9f-0e46-47ce-84a0-0c2dab8bd577","Type":"ContainerStarted","Data":"3b17069cc55ed34fd2c03c9c126ef7a15600927c9be76268a5f3bc88a6f3014f"} Oct 29 23:23:00.728096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:00.727972 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:23:00.784307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:00.784249 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" podStartSLOduration=2.784233412 podStartE2EDuration="2.784233412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:00.780079291 +0000 UTC m=+2377.895998897" watchObservedRunningTime="2026-10-29 23:23:00.784233412 +0000 UTC m=+2377.900152996" Oct 29 23:23:01.209876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:01.209839 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 29 23:23:01.210244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:01.209899 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 29 23:23:02.210512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:02.210478 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 29 23:23:02.210951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:02.210532 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 29 23:23:03.210479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:03.210441 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 29 23:23:03.210683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:03.210510 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 29 23:23:03.781033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:03.780999 5736 generic.go:356] "Generic (PLEG): container finished" podID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerID="faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680" exitCode=0 Oct 29 23:23:03.781271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:03.781066 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" event={"ID":"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666","Type":"ContainerDied","Data":"faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680"} Oct 29 23:23:03.781532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:03.781514 5736 scope.go:122] "RemoveContainer" containerID="faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680" Oct 29 23:23:03.783372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:03.783344 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:23:03.783439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:03.783414 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:23:03.783479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:03.783446 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:23:04.209824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:04.209791 5736 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 29 23:23:04.210001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:04.209841 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 29 23:23:04.792414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:04.792364 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" event={"ID":"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666","Type":"ContainerStarted","Data":"ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb"} Oct 29 23:23:05.216704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:05.216663 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:23:05.217535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:05.217479 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:23:05.218030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:05.218011 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:23:05.434908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:05.434873 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd"] Oct 29 23:23:05.435471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:05.435439 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" containerID="cri-o://7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f" gracePeriod=30 Oct 29 23:23:05.708280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:05.708183 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:23:05.709222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:05.709191 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:23:05.709451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:05.709422 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 29 23:23:06.707242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:06.707209 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:23:06.707673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:06.707267 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 29 23:23:07.706874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:07.706830 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:23:07.707086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:07.706908 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 29 23:23:08.707249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:08.707218 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:23:08.707627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:08.707265 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 29 23:23:09.706607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:09.706542 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 29 23:23:09.706793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:09.706622 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 29 23:23:10.713994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.713963 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:23:10.714530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.714503 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:23:10.715423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.715397 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:23:10.728155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.728096 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:23:10.840595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.840550 5736 generic.go:356] "Generic (PLEG): container finished" podID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerID="7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f" exitCode=0 Oct 29 23:23:10.843213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.843180 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" event={"ID":"47e8d379-c210-485a-85ee-f1c2d00cad24","Type":"ContainerDied","Data":"7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f"} Oct 29 23:23:10.843470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.843442 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" event={"ID":"47e8d379-c210-485a-85ee-f1c2d00cad24","Type":"ContainerDied","Data":"f6dd5c5e68383683cb14f79a963f098eaa21f0fc11a92ed83c782ffb6949da36"} Oct 29 23:23:10.843679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.843485 5736 scope.go:122] "RemoveContainer" containerID="7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f" Oct 29 23:23:10.843913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.843248 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd" Oct 29 23:23:10.896416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.896387 5736 scope.go:122] "RemoveContainer" containerID="7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f" Oct 29 23:23:10.896844 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:23:10.896812 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f\": container with ID starting with 7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f not found: ID does not exist" containerID="7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f" Oct 29 23:23:10.896948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.896855 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f"} err="failed to get container status \"7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f\": rpc error: code = NotFound desc = could not find container \"7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f\": container with ID starting with 7da32b7e4953ebafbbcdf4f488364c1bc7846b979ecd545592ac4b691ca9a34f not found: ID does not exist" Oct 29 23:23:10.923080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.923049 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e8d379-c210-485a-85ee-f1c2d00cad24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e8d379-c210-485a-85ee-f1c2d00cad24-istiod-ca-cert\") pod \"47e8d379-c210-485a-85ee-f1c2d00cad24\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " Oct 29 23:23:10.923234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.923109 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-envoy\") pod \"47e8d379-c210-485a-85ee-f1c2d00cad24\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " Oct 29 23:23:10.923234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.923175 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-podinfo\") pod \"47e8d379-c210-485a-85ee-f1c2d00cad24\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " Oct 29 23:23:10.923352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.923300 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-data\") pod \"47e8d379-c210-485a-85ee-f1c2d00cad24\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " Oct 29 23:23:10.923419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.923366 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-certs\") pod \"47e8d379-c210-485a-85ee-f1c2d00cad24\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " Oct 29 23:23:10.923472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.923451 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-kube-api-access-8b42f\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-kube-api-access-8b42f\") pod \"47e8d379-c210-485a-85ee-f1c2d00cad24\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " Oct 29 23:23:10.923624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.923562 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-credential-socket\") pod \"47e8d379-c210-485a-85ee-f1c2d00cad24\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " Oct 29 23:23:10.923955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.923623 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-token\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-token\") pod \"47e8d379-c210-485a-85ee-f1c2d00cad24\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " Oct 29 23:23:10.925814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.925790 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-socket\") pod \"47e8d379-c210-485a-85ee-f1c2d00cad24\" (UID: \"47e8d379-c210-485a-85ee-f1c2d00cad24\") " Oct 29 23:23:10.926540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.925307 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e8d379-c210-485a-85ee-f1c2d00cad24-istiod-ca-cert" pod "47e8d379-c210-485a-85ee-f1c2d00cad24" (UID: "47e8d379-c210-485a-85ee-f1c2d00cad24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:10.926870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.926849 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e8d379-c210-485a-85ee-f1c2d00cad24-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:10.927413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.925498 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-data" pod "47e8d379-c210-485a-85ee-f1c2d00cad24" (UID: "47e8d379-c210-485a-85ee-f1c2d00cad24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:10.928062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.925595 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-credential-socket" pod "47e8d379-c210-485a-85ee-f1c2d00cad24" (UID: "47e8d379-c210-485a-85ee-f1c2d00cad24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:10.928062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.925632 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-certs" pod "47e8d379-c210-485a-85ee-f1c2d00cad24" (UID: "47e8d379-c210-485a-85ee-f1c2d00cad24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:10.928221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.926507 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-socket" pod "47e8d379-c210-485a-85ee-f1c2d00cad24" (UID: "47e8d379-c210-485a-85ee-f1c2d00cad24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:10.928372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.928307 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-kube-api-access-8b42f" pod "47e8d379-c210-485a-85ee-f1c2d00cad24" (UID: "47e8d379-c210-485a-85ee-f1c2d00cad24"). InnerVolumeSpecName "kube-api-access-8b42f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:10.937026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.936986 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-podinfo" pod "47e8d379-c210-485a-85ee-f1c2d00cad24" (UID: "47e8d379-c210-485a-85ee-f1c2d00cad24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:23:10.937133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.937024 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-token" pod "47e8d379-c210-485a-85ee-f1c2d00cad24" (UID: "47e8d379-c210-485a-85ee-f1c2d00cad24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:10.937706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:10.937682 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-envoy" pod "47e8d379-c210-485a-85ee-f1c2d00cad24" (UID: "47e8d379-c210-485a-85ee-f1c2d00cad24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:11.028332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.028298 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:11.028332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.028333 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:11.028570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.028346 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:11.028570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.028360 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:11.028570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.028374 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b42f\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-kube-api-access-8b42f\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:11.028570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.028387 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:11.028570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.028411 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e8d379-c210-485a-85ee-f1c2d00cad24-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:11.028570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.028424 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e8d379-c210-485a-85ee-f1c2d00cad24-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:11.197362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.197304 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd"] Oct 29 23:23:11.201252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.201221 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-gvbdd"] Oct 29 23:23:11.755073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:11.755041 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:23:13.005867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:13.005836 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" path="/var/lib/kubelet/pods/47e8d379-c210-485a-85ee-f1c2d00cad24/volumes" Oct 29 23:23:13.217186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:13.217131 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z"] Oct 29 23:23:13.217447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:13.217420 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" containerID="cri-o://cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8" gracePeriod=30 Oct 29 23:23:17.256516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.256473 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr"] Oct 29 23:23:17.257375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.257355 5736 cpu_manager.go:414] "containerMap: removing container" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" Oct 29 23:23:17.257375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.257376 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" Oct 29 23:23:17.257492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.257482 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e8d379-c210-485a-85ee-f1c2d00cad24" containerName="istio-proxy" Oct 29 23:23:17.261061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.261035 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.285933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.285899 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e10975-serving-cert\"" Oct 29 23:23:17.290855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.290831 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr"] Oct 29 23:23:17.338169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.338130 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10608824-0cc9-41b0-b4a9-a8230d797de3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.338370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.338211 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcxwc\" (UniqueName: \"kubernetes.io/projected/10608824-0cc9-41b0-b4a9-a8230d797de3-kube-api-access-pcxwc\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.338370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.338252 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.338370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.338311 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-metrics-certs\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.439822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.439787 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10608824-0cc9-41b0-b4a9-a8230d797de3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.439998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.439840 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcxwc\" (UniqueName: \"kubernetes.io/projected/10608824-0cc9-41b0-b4a9-a8230d797de3-kube-api-access-pcxwc\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.439998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.439874 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.439998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.439912 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-metrics-certs\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.440533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.440504 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10608824-0cc9-41b0-b4a9-a8230d797de3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.442352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.442332 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.442520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.442498 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-metrics-certs\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.452155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.452128 5736 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:23:17.452259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.452182 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:23:17.492065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.488147 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcxwc\" (UniqueName: \"kubernetes.io/projected/10608824-0cc9-41b0-b4a9-a8230d797de3-kube-api-access-pcxwc\") pod \"maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.575816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.575732 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:17.885117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.885039 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr"] Oct 29 23:23:17.888808 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:23:17.888774 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10608824_0cc9_41b0_b4a9_a8230d797de3.slice/crio-42a3731146bbe6a063650c66fff1b94a60f4977a0a7739114827e3551fd192e5 WatchSource:0}: Error finding container 42a3731146bbe6a063650c66fff1b94a60f4977a0a7739114827e3551fd192e5: Status 404 returned error can't find the container with id 42a3731146bbe6a063650c66fff1b94a60f4977a0a7739114827e3551fd192e5 Oct 29 23:23:17.890422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.890402 5736 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:23:17.907959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:17.907925 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" event={"ID":"10608824-0cc9-41b0-b4a9-a8230d797de3","Type":"ContainerStarted","Data":"42a3731146bbe6a063650c66fff1b94a60f4977a0a7739114827e3551fd192e5"} Oct 29 23:23:18.603671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.603643 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 23:23:18.719308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719233 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-kube-api-access-xnmqv\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-kube-api-access-xnmqv\") pod \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " Oct 29 23:23:18.719481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719333 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-certs\") pod \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " Oct 29 23:23:18.719481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719379 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istiod-ca-cert\") pod \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " Oct 29 23:23:18.719481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719430 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-data\") pod \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " Oct 29 23:23:18.719682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719542 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-credential-socket\") pod \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " Oct 29 23:23:18.719744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719670 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-certs" pod "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" (UID: "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:18.719802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719739 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-token\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-token\") pod \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " Oct 29 23:23:18.719802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719775 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-envoy\") pod \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " Oct 29 23:23:18.719918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719804 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-podinfo\") pod \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " Oct 29 23:23:18.719918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719806 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-data" pod "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" (UID: "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:18.719918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719851 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-credential-socket" pod "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" (UID: "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:18.719918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719803 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istiod-ca-cert" pod "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" (UID: "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:18.719918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.719888 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-socket\") pod \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\" (UID: \"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4\") " Oct 29 23:23:18.720199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.720147 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-socket" pod "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" (UID: "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:18.720880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.720793 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:18.720880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.720820 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:18.720880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.720835 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:18.720880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.720847 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:18.720880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.720858 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:18.721965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.721898 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-kube-api-access-xnmqv" pod "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" (UID: "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4"). InnerVolumeSpecName "kube-api-access-xnmqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:18.722674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.722648 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-envoy" pod "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" (UID: "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:18.722674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.722660 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-podinfo" pod "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" (UID: "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:23:18.722872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.722792 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-token" pod "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" (UID: "9fe95934-f0ad-421f-b9c1-9c1e6393c5d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:18.822012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.821987 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:18.822012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.822013 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:18.822012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.822025 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:18.822251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.822033 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmqv\" (UniqueName: \"kubernetes.io/projected/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4-kube-api-access-xnmqv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:18.918946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.918871 5736 generic.go:356] "Generic (PLEG): container finished" podID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerID="cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8" exitCode=0 Oct 29 23:23:18.919366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.918958 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" event={"ID":"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4","Type":"ContainerDied","Data":"cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8"} Oct 29 23:23:18.919366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.918926 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" Oct 29 23:23:18.919366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.919028 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z" event={"ID":"9fe95934-f0ad-421f-b9c1-9c1e6393c5d4","Type":"ContainerDied","Data":"5dfd5d60b2f74f3f6d645ed34b5315af084af58e4ed96c06a36c170f5ec3ac3c"} Oct 29 23:23:18.919366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.919061 5736 scope.go:122] "RemoveContainer" containerID="cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8" Oct 29 23:23:18.921503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.921480 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" event={"ID":"10608824-0cc9-41b0-b4a9-a8230d797de3","Type":"ContainerStarted","Data":"a879deccc3f4d296ac49fb7ad77ad42beda41bf56866fb63c4bde7ec6a76eeb5"} Oct 29 23:23:18.921677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.921542 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:18.931879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.931857 5736 scope.go:122] "RemoveContainer" containerID="cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8" Oct 29 23:23:18.932185 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:23:18.932164 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8\": container with ID starting with cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8 not found: ID does not exist" containerID="cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8" Oct 29 23:23:18.932247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.932192 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8"} err="failed to get container status \"cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8\": rpc error: code = NotFound desc = could not find container \"cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8\": container with ID starting with cd646b13b39f9823aac0ad4270ca70c34a310288f8d6ddba3a7322b2a69e3be8 not found: ID does not exist" Oct 29 23:23:18.967623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.967598 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z"] Oct 29 23:23:18.977096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:18.977043 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-rd84z"] Oct 29 23:23:19.012717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:19.012662 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" podStartSLOduration=1.708094502 podStartE2EDuration="2.012648503s" totalImagesPullingTime="304.554001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:19.010159141 +0000 UTC m=+2396.126078747" watchObservedRunningTime="2026-10-29 23:23:19.012648503 +0000 UTC m=+2396.128568108" Oct 29 23:23:19.016371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:19.016330 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" path="/var/lib/kubelet/pods/9fe95934-f0ad-421f-b9c1-9c1e6393c5d4/volumes" Oct 29 23:23:23.838274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:23.838244 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:23:23.864453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:23.864428 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:23:24.396602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.390492 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv"] Oct 29 23:23:24.396602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.394335 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" Oct 29 23:23:24.396602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.394362 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" Oct 29 23:23:24.396602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.394570 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe95934-f0ad-421f-b9c1-9c1e6393c5d4" containerName="istio-proxy" Oct 29 23:23:24.434601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.417863 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.434601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.433355 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv"] Oct 29 23:23:24.443840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.443645 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c5ef2f-serving-cert\"" Oct 29 23:23:24.508660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.508619 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.508958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.508932 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-metrics-certs\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.509111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.508985 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwm4\" (UniqueName: \"kubernetes.io/projected/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-kube-api-access-zgwm4\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.509111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.509058 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.610809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.610761 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-metrics-certs\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.611174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.611141 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwm4\" (UniqueName: \"kubernetes.io/projected/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-kube-api-access-zgwm4\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.611389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.611190 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.611389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.611261 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.611656 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:23:24.611633 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-serving-cert: secret "maas-api-e2e-extmod-a-c5ef2f-serving-cert" not found Oct 29 23:23:24.611751 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:23:24.611708 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls podName:d8dbbcec-24d4-4f48-a086-c62bc4e1d379 nodeName:}" failed. No retries permitted until 2026-10-29 23:23:25.111686511 +0000 UTC m=+2402.227606096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls") pod "maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" (UID: "d8dbbcec-24d4-4f48-a086-c62bc4e1d379") : secret "maas-api-e2e-extmod-a-c5ef2f-serving-cert" not found Oct 29 23:23:24.612047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.612017 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.613863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.613830 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-metrics-certs\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.632193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.632164 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwm4\" (UniqueName: \"kubernetes.io/projected/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-kube-api-access-zgwm4\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:24.935934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:24.935907 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:23:25.118862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:25.118807 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:25.121285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:25.121252 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:25.348519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:25.348488 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:25.877551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:25.877520 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv"] Oct 29 23:23:25.878535 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:23:25.878506 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8dbbcec_24d4_4f48_a086_c62bc4e1d379.slice/crio-b7d74161773ef6e8b2128614d9a9dab513f0cfb55bb4c058d21ff60e4ae447c9 WatchSource:0}: Error finding container b7d74161773ef6e8b2128614d9a9dab513f0cfb55bb4c058d21ff60e4ae447c9: Status 404 returned error can't find the container with id b7d74161773ef6e8b2128614d9a9dab513f0cfb55bb4c058d21ff60e4ae447c9 Oct 29 23:23:25.972783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:25.972747 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" event={"ID":"d8dbbcec-24d4-4f48-a086-c62bc4e1d379","Type":"ContainerStarted","Data":"b7d74161773ef6e8b2128614d9a9dab513f0cfb55bb4c058d21ff60e4ae447c9"} Oct 29 23:23:26.826473 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:23:26.826442 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:23:26.981394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:26.981353 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" event={"ID":"d8dbbcec-24d4-4f48-a086-c62bc4e1d379","Type":"ContainerStarted","Data":"40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18"} Oct 29 23:23:26.981823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:26.981622 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:27.022444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:27.022388 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" podStartSLOduration=2.684269353 podStartE2EDuration="3.022373738s" totalImagesPullingTime="338.104385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:27.01846966 +0000 UTC m=+2404.134389265" watchObservedRunningTime="2026-10-29 23:23:27.022373738 +0000 UTC m=+2404.138293343" Oct 29 23:23:31.372836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.372802 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq"] Oct 29 23:23:31.378903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.378882 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.391928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.391898 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:23:31.396845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.396817 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq"] Oct 29 23:23:31.405637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.403369 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:23:31.470771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.470737 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcq2s\" (UniqueName: \"kubernetes.io/projected/32a08b3d-3db1-40ab-a093-10f3046bcf46-kube-api-access-gcq2s\") pod \"maas-api-revoke-sub-9905f98a-8fdnq\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.470974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.470944 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9905f98a-8fdnq\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.471073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.471056 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9905f98a-8fdnq\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.572618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.572564 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9905f98a-8fdnq\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.572813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.572665 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9905f98a-8fdnq\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.572885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.572819 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcq2s\" (UniqueName: \"kubernetes.io/projected/32a08b3d-3db1-40ab-a093-10f3046bcf46-kube-api-access-gcq2s\") pod \"maas-api-revoke-sub-9905f98a-8fdnq\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.573509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.573480 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9905f98a-8fdnq\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.574978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.574958 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9905f98a-8fdnq\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.595273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.595238 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcq2s\" (UniqueName: \"kubernetes.io/projected/32a08b3d-3db1-40ab-a093-10f3046bcf46-kube-api-access-gcq2s\") pod \"maas-api-revoke-sub-9905f98a-8fdnq\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:31.711659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:31.698245 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:32.181105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:32.181076 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq"] Oct 29 23:23:32.183518 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:23:32.183488 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a08b3d_3db1_40ab_a093_10f3046bcf46.slice/crio-ea4f6c525647ba6db3fc56a18431f521965910563ef6a34eb9bf2cb9a86ba217 WatchSource:0}: Error finding container ea4f6c525647ba6db3fc56a18431f521965910563ef6a34eb9bf2cb9a86ba217: Status 404 returned error can't find the container with id ea4f6c525647ba6db3fc56a18431f521965910563ef6a34eb9bf2cb9a86ba217 Oct 29 23:23:32.994822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:32.994791 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:23:33.033265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:33.033214 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" event={"ID":"32a08b3d-3db1-40ab-a093-10f3046bcf46","Type":"ContainerStarted","Data":"8775832e1b5315d02cc73fb6571a78c02250ff02d558c625c540fdd8aa8a695f"} Oct 29 23:23:33.033434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:33.033301 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" event={"ID":"32a08b3d-3db1-40ab-a093-10f3046bcf46","Type":"ContainerStarted","Data":"ea4f6c525647ba6db3fc56a18431f521965910563ef6a34eb9bf2cb9a86ba217"} Oct 29 23:23:33.085787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:33.085724 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" podStartSLOduration=2.085705034 podStartE2EDuration="2.085705034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:33.078776855 +0000 UTC m=+2410.194696461" watchObservedRunningTime="2026-10-29 23:23:33.085705034 +0000 UTC m=+2410.201624641" Oct 29 23:23:34.030795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:34.030758 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm"] Oct 29 23:23:34.042554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:34.042519 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c8ddf8-v74jm"] Oct 29 23:23:34.046888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:34.046862 5736 generic.go:356] "Generic (PLEG): container finished" podID="32a08b3d-3db1-40ab-a093-10f3046bcf46" containerID="8775832e1b5315d02cc73fb6571a78c02250ff02d558c625c540fdd8aa8a695f" exitCode=0 Oct 29 23:23:34.047034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:34.046906 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" event={"ID":"32a08b3d-3db1-40ab-a093-10f3046bcf46","Type":"ContainerDied","Data":"8775832e1b5315d02cc73fb6571a78c02250ff02d558c625c540fdd8aa8a695f"} Oct 29 23:23:35.005555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.005518 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f82ec59-ff49-45ed-ad06-c7c88ced15f0" path="/var/lib/kubelet/pods/5f82ec59-ff49-45ed-ad06-c7c88ced15f0/volumes" Oct 29 23:23:35.218383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.218357 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:35.253774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.253747 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-openshift-ca\") pod \"32a08b3d-3db1-40ab-a093-10f3046bcf46\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " Oct 29 23:23:35.253942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.253913 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a08b3d-3db1-40ab-a093-10f3046bcf46-kube-api-access-gcq2s\" (UniqueName: \"kubernetes.io/projected/32a08b3d-3db1-40ab-a093-10f3046bcf46-kube-api-access-gcq2s\") pod \"32a08b3d-3db1-40ab-a093-10f3046bcf46\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " Oct 29 23:23:35.254016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.254009 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-xks-ca\") pod \"32a08b3d-3db1-40ab-a093-10f3046bcf46\" (UID: \"32a08b3d-3db1-40ab-a093-10f3046bcf46\") " Oct 29 23:23:35.254156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.254131 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-openshift-ca" pod "32a08b3d-3db1-40ab-a093-10f3046bcf46" (UID: "32a08b3d-3db1-40ab-a093-10f3046bcf46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:35.254502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.254482 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:35.256038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.255987 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a08b3d-3db1-40ab-a093-10f3046bcf46-kube-api-access-gcq2s" pod "32a08b3d-3db1-40ab-a093-10f3046bcf46" (UID: "32a08b3d-3db1-40ab-a093-10f3046bcf46"). InnerVolumeSpecName "kube-api-access-gcq2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:35.256038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.255987 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-xks-ca" pod "32a08b3d-3db1-40ab-a093-10f3046bcf46" (UID: "32a08b3d-3db1-40ab-a093-10f3046bcf46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:35.355448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.355423 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcq2s\" (UniqueName: \"kubernetes.io/projected/32a08b3d-3db1-40ab-a093-10f3046bcf46-kube-api-access-gcq2s\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:35.355448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:35.355448 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a08b3d-3db1-40ab-a093-10f3046bcf46-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:36.087196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:36.087161 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" event={"ID":"32a08b3d-3db1-40ab-a093-10f3046bcf46","Type":"ContainerDied","Data":"ea4f6c525647ba6db3fc56a18431f521965910563ef6a34eb9bf2cb9a86ba217"} Oct 29 23:23:36.087412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:36.087220 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea4f6c525647ba6db3fc56a18431f521965910563ef6a34eb9bf2cb9a86ba217" Oct 29 23:23:36.087412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:36.087286 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq" Oct 29 23:23:37.521763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.521728 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j"] Oct 29 23:23:37.522655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.522633 5736 cpu_manager.go:414] "containerMap: removing container" podUID="32a08b3d-3db1-40ab-a093-10f3046bcf46" containerName="revoke-keys" Oct 29 23:23:37.522655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.522655 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a08b3d-3db1-40ab-a093-10f3046bcf46" containerName="revoke-keys" Oct 29 23:23:37.522811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.522781 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a08b3d-3db1-40ab-a093-10f3046bcf46" containerName="revoke-keys" Oct 29 23:23:37.526525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.526508 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:37.545339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.545311 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j"] Oct 29 23:23:37.551798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.551773 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4e2ac7-dockercfg-kvj79\"" Oct 29 23:23:37.551937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.551915 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4e2ac7\"" Oct 29 23:23:37.584012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.583981 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9"] Oct 29 23:23:37.589331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.589306 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:37.589666 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.589643 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzf7k\" (UniqueName: \"kubernetes.io/projected/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-kube-api-access-xzf7k\") pod \"payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j\" (UID: \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:37.589849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.589830 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j\" (UID: \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:37.606782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.606757 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9"] Oct 29 23:23:37.692231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.692192 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzf7k\" (UniqueName: \"kubernetes.io/projected/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-kube-api-access-xzf7k\") pod \"payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j\" (UID: \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:37.692540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.692410 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hp6w\" (UniqueName: \"kubernetes.io/projected/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-kube-api-access-2hp6w\") pod \"payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9\" (UID: \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:37.692540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.692484 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9\" (UID: \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:37.692540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.692529 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j\" (UID: \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:37.694774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.694748 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j\" (UID: \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:37.741921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.741882 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzf7k\" (UniqueName: \"kubernetes.io/projected/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-kube-api-access-xzf7k\") pod \"payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j\" (UID: \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:37.794458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.794378 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hp6w\" (UniqueName: \"kubernetes.io/projected/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-kube-api-access-2hp6w\") pod \"payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9\" (UID: \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:37.794654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.794469 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9\" (UID: \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:37.795487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.795454 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9\" (UID: \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:37.816198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.816161 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hp6w\" (UniqueName: \"kubernetes.io/projected/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-kube-api-access-2hp6w\") pod \"payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9\" (UID: \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:37.840888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.840853 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:37.916273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:37.903302 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:38.183954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:38.183930 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j"] Oct 29 23:23:38.186815 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:23:38.186786 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca615e7b_9e24_4aa4_a005_bbd24c17a39a.slice/crio-1ec4775c194385b84c936651c2ae367667ed01d5b78ba1e77db1c2dfcb7f22f2 WatchSource:0}: Error finding container 1ec4775c194385b84c936651c2ae367667ed01d5b78ba1e77db1c2dfcb7f22f2: Status 404 returned error can't find the container with id 1ec4775c194385b84c936651c2ae367667ed01d5b78ba1e77db1c2dfcb7f22f2 Oct 29 23:23:38.292324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:38.292298 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9"] Oct 29 23:23:38.294653 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:23:38.294623 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7efa29c_fb60_46e2_baa2_15d5c816a4e7.slice/crio-fb431fcc603661c92ad946837640bd5ba640cc574615caa5bae7f287d460bc9f WatchSource:0}: Error finding container fb431fcc603661c92ad946837640bd5ba640cc574615caa5bae7f287d460bc9f: Status 404 returned error can't find the container with id fb431fcc603661c92ad946837640bd5ba640cc574615caa5bae7f287d460bc9f Oct 29 23:23:39.114846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:39.114812 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" event={"ID":"ca615e7b-9e24-4aa4-a005-bbd24c17a39a","Type":"ContainerStarted","Data":"9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49"} Oct 29 23:23:39.115287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:39.114875 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:39.115287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:39.114904 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" event={"ID":"ca615e7b-9e24-4aa4-a005-bbd24c17a39a","Type":"ContainerStarted","Data":"1ec4775c194385b84c936651c2ae367667ed01d5b78ba1e77db1c2dfcb7f22f2"} Oct 29 23:23:39.116464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:39.116429 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" event={"ID":"a7efa29c-fb60-46e2-baa2-15d5c816a4e7","Type":"ContainerStarted","Data":"9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4"} Oct 29 23:23:39.116631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:39.116487 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:39.116631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:39.116510 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" event={"ID":"a7efa29c-fb60-46e2-baa2-15d5c816a4e7","Type":"ContainerStarted","Data":"fb431fcc603661c92ad946837640bd5ba640cc574615caa5bae7f287d460bc9f"} Oct 29 23:23:39.153718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:39.153673 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" podStartSLOduration=2.153659507 podStartE2EDuration="2.153659507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:39.151647945 +0000 UTC m=+2416.267567552" watchObservedRunningTime="2026-10-29 23:23:39.153659507 +0000 UTC m=+2416.269579113" Oct 29 23:23:39.191206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:39.191157 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" podStartSLOduration=2.1911434 podStartE2EDuration="2.1911434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:39.188105897 +0000 UTC m=+2416.304025525" watchObservedRunningTime="2026-10-29 23:23:39.1911434 +0000 UTC m=+2416.307063006" Oct 29 23:23:42.882689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:42.882652 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f"] Oct 29 23:23:47.883730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:47.883682 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" podUID="cdc7de1d-080d-4700-9a10-88583217467f" containerName="payload-processing" containerID="cri-o://870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0" gracePeriod=25 Oct 29 23:23:48.162120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.162095 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:23:48.190804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.190777 5736 generic.go:356] "Generic (PLEG): container finished" podID="cdc7de1d-080d-4700-9a10-88583217467f" containerID="870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0" exitCode=0 Oct 29 23:23:48.190940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.190819 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" Oct 29 23:23:48.190940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.190863 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" event={"ID":"cdc7de1d-080d-4700-9a10-88583217467f","Type":"ContainerDied","Data":"870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0"} Oct 29 23:23:48.190940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.190918 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f" event={"ID":"cdc7de1d-080d-4700-9a10-88583217467f","Type":"ContainerDied","Data":"10daac6c5b45bf730d5b1f5d0d049babbcb797b7fb166d55b1d1165bd825d3b8"} Oct 29 23:23:48.190940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.190940 5736 scope.go:122] "RemoveContainer" containerID="870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0" Oct 29 23:23:48.201214 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.200757 5736 scope.go:122] "RemoveContainer" containerID="870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0" Oct 29 23:23:48.201382 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:23:48.201349 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0\": container with ID starting with 870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0 not found: ID does not exist" containerID="870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0" Oct 29 23:23:48.201881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.201392 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0"} err="failed to get container status \"870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0\": rpc error: code = NotFound desc = could not find container \"870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0\": container with ID starting with 870957bedfb8d6cb5a4a2becde07d9c3447eaba14c047f9cfa2193050b9e04f0 not found: ID does not exist" Oct 29 23:23:48.273943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.273919 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-plugins-config-volume\") pod \"cdc7de1d-080d-4700-9a10-88583217467f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " Oct 29 23:23:48.274105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.274060 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-provider-credentials\") pod \"cdc7de1d-080d-4700-9a10-88583217467f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " Oct 29 23:23:48.274105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.274093 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-routing-overlay-volume\") pod \"cdc7de1d-080d-4700-9a10-88583217467f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " Oct 29 23:23:48.274237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.274137 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-kube-api-access-ddnww\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-kube-api-access-ddnww\") pod \"cdc7de1d-080d-4700-9a10-88583217467f\" (UID: \"cdc7de1d-080d-4700-9a10-88583217467f\") " Oct 29 23:23:48.274427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.274313 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-plugins-config-volume" pod "cdc7de1d-080d-4700-9a10-88583217467f" (UID: "cdc7de1d-080d-4700-9a10-88583217467f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:48.274508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.274486 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-routing-overlay-volume" pod "cdc7de1d-080d-4700-9a10-88583217467f" (UID: "cdc7de1d-080d-4700-9a10-88583217467f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:23:48.274802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.274776 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:48.274802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.274805 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cdc7de1d-080d-4700-9a10-88583217467f-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:48.276450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.276427 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-provider-credentials" pod "cdc7de1d-080d-4700-9a10-88583217467f" (UID: "cdc7de1d-080d-4700-9a10-88583217467f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:48.276522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.276458 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-kube-api-access-ddnww" pod "cdc7de1d-080d-4700-9a10-88583217467f" (UID: "cdc7de1d-080d-4700-9a10-88583217467f"). InnerVolumeSpecName "kube-api-access-ddnww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:48.376425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.376394 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddnww\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-kube-api-access-ddnww\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:48.376425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.376420 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cdc7de1d-080d-4700-9a10-88583217467f-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:23:48.546383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.546348 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f"] Oct 29 23:23:48.556051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:48.556019 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-26b322/payload-processing-external-model-e2e-extmod-b-26b322-54c9m5r9f"] Oct 29 23:23:48.559810 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:23:48.559783 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc7de1d_080d_4700_9a10_88583217467f.slice/crio-10daac6c5b45bf730d5b1f5d0d049babbcb797b7fb166d55b1d1165bd825d3b8\": RecentStats: unable to find data in memory cache]" Oct 29 23:23:49.056141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:49.054147 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc7de1d-080d-4700-9a10-88583217467f" path="/var/lib/kubelet/pods/cdc7de1d-080d-4700-9a10-88583217467f/volumes" Oct 29 23:23:50.127089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:50.127055 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:23:50.127567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:50.127119 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:23:56.825279 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:23:56.825242 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:23:59.193802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.193771 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h"] Oct 29 23:23:59.194488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.194457 5736 cpu_manager.go:414] "containerMap: removing container" podUID="cdc7de1d-080d-4700-9a10-88583217467f" containerName="payload-processing" Oct 29 23:23:59.194488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.194478 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc7de1d-080d-4700-9a10-88583217467f" containerName="payload-processing" Oct 29 23:23:59.194769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.194752 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc7de1d-080d-4700-9a10-88583217467f" containerName="payload-processing" Oct 29 23:23:59.198667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.198647 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.216376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.216353 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dea9cb\"/\"agc-cred-bde07b33cac4\"" Oct 29 23:23:59.216471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.216384 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dea9cb\"/\"payload-processing-external-model-e2e--d7d24801-dockercfg-fmcvd\"" Oct 29 23:23:59.226706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.226671 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dea9cb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dea9cb\"" Oct 29 23:23:59.226706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.226678 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dea9cb\"/\"routing-overlay\"" Oct 29 23:23:59.226706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.226680 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dea9cb\"/\"openshift-service-ca.crt\"" Oct 29 23:23:59.226952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.226680 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dea9cb\"/\"kube-root-ca.crt\"" Oct 29 23:23:59.230928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.230901 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h"] Oct 29 23:23:59.305680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.303746 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.305680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.303843 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.305680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.303898 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.305680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.303973 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f69lj\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-kube-api-access-f69lj\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.405834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.405794 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.406165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.406140 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.406331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.406314 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.406513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.406497 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f69lj\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-kube-api-access-f69lj\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.413351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.413319 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.414103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.414071 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.427653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.419989 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.449602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.443118 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f69lj\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-kube-api-access-f69lj\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.513115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.513083 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:23:59.822613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:23:59.822566 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h"] Oct 29 23:23:59.825886 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:23:59.825854 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd642cf_4946_445f_bd55_315a5703ba80.slice/crio-e40700c5f58588d3ed9e48107b672182b3854c7d04a8e9d1d7755812e3286a00 WatchSource:0}: Error finding container e40700c5f58588d3ed9e48107b672182b3854c7d04a8e9d1d7755812e3286a00: Status 404 returned error can't find the container with id e40700c5f58588d3ed9e48107b672182b3854c7d04a8e9d1d7755812e3286a00 Oct 29 23:24:00.325365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:00.325330 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" event={"ID":"bcd642cf-4946-445f-bd55-315a5703ba80","Type":"ContainerStarted","Data":"1d11bcc53c4924944ef77ea375c05f24f4e60ad0cdff6e908516b8c05e2765a8"} Oct 29 23:24:00.325783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:00.325383 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" event={"ID":"bcd642cf-4946-445f-bd55-315a5703ba80","Type":"ContainerStarted","Data":"e40700c5f58588d3ed9e48107b672182b3854c7d04a8e9d1d7755812e3286a00"} Oct 29 23:24:00.325783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:00.325409 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:24:00.364775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:00.364719 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" podStartSLOduration=1.364706121 podStartE2EDuration="1.364706121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:00.361750897 +0000 UTC m=+2437.477670504" watchObservedRunningTime="2026-10-29 23:24:00.364706121 +0000 UTC m=+2437.480625726" Oct 29 23:24:11.337140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:11.337056 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:24:16.031183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.031148 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz"] Oct 29 23:24:16.036819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.036793 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.062198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.062166 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c1e164\"/\"default-dockercfg-whrvt\"" Oct 29 23:24:16.062481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.062459 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c1e164\"/\"openshift-service-ca.crt\"" Oct 29 23:24:16.062680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.062474 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c1e164\"/\"autores-model-c1e164-kserve-self-signed-certs\"" Oct 29 23:24:16.062811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.062519 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c1e164\"/\"kube-root-ca.crt\"" Oct 29 23:24:16.066784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.066759 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz"] Oct 29 23:24:16.184943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.184905 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-model-cache\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.185232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.185206 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfl77\" (UniqueName: \"kubernetes.io/projected/38d9982f-cecd-44fe-ae74-a8172a361338-kube-api-access-rfl77\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.185432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.185404 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-home\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.185570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.185475 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-dshm\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.185570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.185560 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9982f-cecd-44fe-ae74-a8172a361338-tls-certs\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.185861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.185832 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-tmp-dir\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.287634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.287525 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-model-cache\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.287634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.287588 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfl77\" (UniqueName: \"kubernetes.io/projected/38d9982f-cecd-44fe-ae74-a8172a361338-kube-api-access-rfl77\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.287881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.287647 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-home\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.287881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.287701 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-dshm\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.287881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.287776 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9982f-cecd-44fe-ae74-a8172a361338-tls-certs\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.288066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.287949 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-tmp-dir\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.288066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.288004 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-model-cache\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.288190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.288067 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-home\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.288410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.288386 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-tmp-dir\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.290639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.290570 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-dshm\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.290824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.290806 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9982f-cecd-44fe-ae74-a8172a361338-tls-certs\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.311474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.311449 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfl77\" (UniqueName: \"kubernetes.io/projected/38d9982f-cecd-44fe-ae74-a8172a361338-kube-api-access-rfl77\") pod \"autores-model-c1e164-kserve-75fc747c97-4mbhz\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.361278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.361241 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:16.893911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:16.893883 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz"] Oct 29 23:24:16.896781 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:24:16.896748 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38d9982f_cecd_44fe_ae74_a8172a361338.slice/crio-17f3c027d306fbda70ba8e8cba4fa9461dd517cee162fa70d8482ce476817979 WatchSource:0}: Error finding container 17f3c027d306fbda70ba8e8cba4fa9461dd517cee162fa70d8482ce476817979: Status 404 returned error can't find the container with id 17f3c027d306fbda70ba8e8cba4fa9461dd517cee162fa70d8482ce476817979 Oct 29 23:24:17.793966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:17.793917 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" event={"ID":"38d9982f-cecd-44fe-ae74-a8172a361338","Type":"ContainerStarted","Data":"f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd"} Oct 29 23:24:17.793966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:17.793969 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" event={"ID":"38d9982f-cecd-44fe-ae74-a8172a361338","Type":"ContainerStarted","Data":"17f3c027d306fbda70ba8e8cba4fa9461dd517cee162fa70d8482ce476817979"} Oct 29 23:24:17.833807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:17.833749 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" podStartSLOduration=1.833730909 podStartE2EDuration="1.833730909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:17.830536114 +0000 UTC m=+2454.946455719" watchObservedRunningTime="2026-10-29 23:24:17.833730909 +0000 UTC m=+2454.949650517" Oct 29 23:24:18.362720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:18.362672 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:22.568615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.568557 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh"] Oct 29 23:24:22.573542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.573506 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.588203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.588163 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7d4110-openshift-default-dockercfg-2g5ld\"" Oct 29 23:24:22.603242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.603208 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh"] Oct 29 23:24:22.629657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.629619 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istiod-ca-cert\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.629818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.629731 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-token\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.629818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.629772 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-data\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.629912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.629854 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-socket\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.629977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.629926 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-certs\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.630046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.630018 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-podinfo\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.630239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.630201 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-credential-socket\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.630381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.630251 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrp4\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-kube-api-access-9wrp4\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.630500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.630476 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-envoy\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.732390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732351 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-data\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.732390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732399 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-socket\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.732688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732449 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-certs\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.732688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732475 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-podinfo\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.732688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732565 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-credential-socket\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.732688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732650 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrp4\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-kube-api-access-9wrp4\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.732929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732831 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-envoy\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.732929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732862 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-data\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.733050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732945 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istiod-ca-cert\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.733050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.732962 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-socket\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.733050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.733040 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-token\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.733234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.733050 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-credential-socket\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.733234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.733187 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-certs\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.733678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.733654 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istiod-ca-cert\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.735012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.734988 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-envoy\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.735486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.735468 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-podinfo\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.754703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.754671 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrp4\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-kube-api-access-9wrp4\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.754831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.754730 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-token\") pod \"e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:22.896685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:22.896352 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:23.543443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:23.543405 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh"] Oct 29 23:24:23.543611 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:24:23.543550 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ee0caa_a5e0_4c18_81c5_0e675a4ba9ac.slice/crio-35f12b713dde5e7c6b64bc82fa7c69859d1049f2581e24fb74058014c4084bbb WatchSource:0}: Error finding container 35f12b713dde5e7c6b64bc82fa7c69859d1049f2581e24fb74058014c4084bbb: Status 404 returned error can't find the container with id 35f12b713dde5e7c6b64bc82fa7c69859d1049f2581e24fb74058014c4084bbb Oct 29 23:24:23.547573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:23.547531 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:24:23.547678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:23.547636 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:24:23.547726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:23.547680 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:24:23.947075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:23.946984 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" event={"ID":"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac","Type":"ContainerStarted","Data":"37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8"} Oct 29 23:24:23.948631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:23.947641 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" event={"ID":"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac","Type":"ContainerStarted","Data":"35f12b713dde5e7c6b64bc82fa7c69859d1049f2581e24fb74058014c4084bbb"} Oct 29 23:24:24.015756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:24.015694 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" podStartSLOduration=2.015680065 podStartE2EDuration="2.015680065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:24.005188974 +0000 UTC m=+2461.121108579" watchObservedRunningTime="2026-10-29 23:24:24.015680065 +0000 UTC m=+2461.131599670" Oct 29 23:24:24.901386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:24.901347 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:24.923236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:24.923204 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:24.976616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:24.976572 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:24.978200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:24.978177 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:24:26.365602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:26.364669 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:26.379129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:26.379100 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:26.820658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:26.817716 5736 scope.go:122] "RemoveContainer" containerID="5e117547f7b88f244bef0cd73055a6c01477173406a229fe101ff35448695844" Oct 29 23:24:26.913875 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:24:26.913840 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:24:27.006595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:27.006551 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:24:28.795602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:28.786664 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth"] Oct 29 23:24:28.795602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:28.793917 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:28.821604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:28.808938 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7d4110-dockercfg-fdkv4\"" Oct 29 23:24:28.870893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:28.861917 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7d4110\"" Oct 29 23:24:28.885598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:28.881442 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth"] Oct 29 23:24:28.924665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:28.920986 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdrg\" (UniqueName: \"kubernetes.io/projected/1fcf1dc2-3f96-49df-9d31-a838a398ef06-kube-api-access-hzdrg\") pod \"payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth\" (UID: \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:28.924665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:28.921355 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf1dc2-3f96-49df-9d31-a838a398ef06-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth\" (UID: \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:29.022661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:29.022627 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf1dc2-3f96-49df-9d31-a838a398ef06-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth\" (UID: \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:29.022863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:29.022770 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdrg\" (UniqueName: \"kubernetes.io/projected/1fcf1dc2-3f96-49df-9d31-a838a398ef06-kube-api-access-hzdrg\") pod \"payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth\" (UID: \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:29.023565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:29.023535 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf1dc2-3f96-49df-9d31-a838a398ef06-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth\" (UID: \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:29.059713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:29.059544 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdrg\" (UniqueName: \"kubernetes.io/projected/1fcf1dc2-3f96-49df-9d31-a838a398ef06-kube-api-access-hzdrg\") pod \"payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth\" (UID: \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:29.314434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:29.314350 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:29.659155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:29.659119 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth"] Oct 29 23:24:30.121665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:30.121546 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" event={"ID":"1fcf1dc2-3f96-49df-9d31-a838a398ef06","Type":"ContainerStarted","Data":"f473b5ce486a3464af5b2bf8003cf2f2daae58936f257090abfa23c57dc3d8be"} Oct 29 23:24:31.135625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:31.135538 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" event={"ID":"1fcf1dc2-3f96-49df-9d31-a838a398ef06","Type":"ContainerStarted","Data":"7697da753d9347e1186a24ba39acd495be8b275331ca9932aaf0d4bc3caffb91"} Oct 29 23:24:31.136183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:31.135803 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:31.176833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:31.176769 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" podStartSLOduration=3.176748043 podStartE2EDuration="3.176748043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:31.175330909 +0000 UTC m=+2468.291250516" watchObservedRunningTime="2026-10-29 23:24:31.176748043 +0000 UTC m=+2468.292667649" Oct 29 23:24:34.162765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:34.162682 5736 generic.go:356] "Generic (PLEG): container finished" podID="b04a1403-813a-492d-8472-978da4638bad" containerID="a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01" exitCode=0 Oct 29 23:24:34.162765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:34.162725 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" event={"ID":"b04a1403-813a-492d-8472-978da4638bad","Type":"ContainerDied","Data":"a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01"} Oct 29 23:24:34.163360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:34.163289 5736 scope.go:122] "RemoveContainer" containerID="a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01" Oct 29 23:24:34.165509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:34.165474 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:24:34.165648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:34.165546 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:24:34.165648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:34.165595 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:24:35.172843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:35.172806 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" event={"ID":"b04a1403-813a-492d-8472-978da4638bad","Type":"ContainerStarted","Data":"4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d"} Oct 29 23:24:35.365616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:35.363059 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:24:35.375863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:35.374802 5736 patch_prober.go:29] interesting pod/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 29 23:24:35.375863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:35.375023 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 29 23:24:36.375322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:36.365003 5736 patch_prober.go:29] interesting pod/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 29 23:24:36.375322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:36.365074 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 29 23:24:37.350992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:37.350951 5736 patch_prober.go:29] interesting pod/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 29 23:24:37.351301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:37.351266 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 29 23:24:37.689675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:37.685920 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l"] Oct 29 23:24:37.707254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:37.691803 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:37.766614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:37.747073 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d5e482\"" Oct 29 23:24:37.766614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:37.747154 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d5e482-dockercfg-x5488\"" Oct 29 23:24:37.830680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:37.830638 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l"] Oct 29 23:24:37.973748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:37.973709 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604aee36-959a-4dd8-8b9c-108e22cf2649-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l\" (UID: \"604aee36-959a-4dd8-8b9c-108e22cf2649\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:37.974166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:37.974149 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqglx\" (UniqueName: \"kubernetes.io/projected/604aee36-959a-4dd8-8b9c-108e22cf2649-kube-api-access-lqglx\") pod \"payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l\" (UID: \"604aee36-959a-4dd8-8b9c-108e22cf2649\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:38.085411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:38.085265 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604aee36-959a-4dd8-8b9c-108e22cf2649-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l\" (UID: \"604aee36-959a-4dd8-8b9c-108e22cf2649\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:38.085627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:38.085486 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqglx\" (UniqueName: \"kubernetes.io/projected/604aee36-959a-4dd8-8b9c-108e22cf2649-kube-api-access-lqglx\") pod \"payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l\" (UID: \"604aee36-959a-4dd8-8b9c-108e22cf2649\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:38.086518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:38.086490 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604aee36-959a-4dd8-8b9c-108e22cf2649-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l\" (UID: \"604aee36-959a-4dd8-8b9c-108e22cf2649\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:38.114927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:38.114892 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqglx\" (UniqueName: \"kubernetes.io/projected/604aee36-959a-4dd8-8b9c-108e22cf2649-kube-api-access-lqglx\") pod \"payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l\" (UID: \"604aee36-959a-4dd8-8b9c-108e22cf2649\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:38.362793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:38.356530 5736 patch_prober.go:29] interesting pod/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 29 23:24:38.362793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:38.356626 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 29 23:24:38.377168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:38.377130 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:38.780021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:38.763457 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l"] Oct 29 23:24:39.207782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:39.207696 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" event={"ID":"604aee36-959a-4dd8-8b9c-108e22cf2649","Type":"ContainerStarted","Data":"96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7"} Oct 29 23:24:39.207782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:39.207768 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:39.207970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:39.207794 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" event={"ID":"604aee36-959a-4dd8-8b9c-108e22cf2649","Type":"ContainerStarted","Data":"f19e81d1cac018a4693af7bd775a58ec24d0d21ce17eb47f2237b673a6219f09"} Oct 29 23:24:39.350603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:39.350010 5736 patch_prober.go:29] interesting pod/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 29 23:24:39.350603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:39.350080 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 29 23:24:40.355956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:40.355918 5736 patch_prober.go:29] interesting pod/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 29 23:24:40.356499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:40.355977 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 29 23:24:41.355535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:41.355504 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:24:41.356012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:41.355995 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:24:41.356721 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:41.356696 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:24:41.418104 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:41.418051 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" podStartSLOduration=4.418038348 podStartE2EDuration="4.418038348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:39.255888515 +0000 UTC m=+2476.371808122" watchObservedRunningTime="2026-10-29 23:24:41.418038348 +0000 UTC m=+2478.533957954" Oct 29 23:24:42.146451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:42.146421 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:24:50.218760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:50.218729 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:24:53.148954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.148902 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz"] Oct 29 23:24:53.503114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.503083 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d"] Oct 29 23:24:53.507741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.507714 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.522813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.522787 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:24:53.523004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.522911 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:24:53.528670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.528639 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d"] Oct 29 23:24:53.587106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.587072 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c1e164-d5g9d\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.587386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.587361 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwz27\" (UniqueName: \"kubernetes.io/projected/4df618e7-36a4-4859-b21b-0400e64d5f2c-kube-api-access-rwz27\") pod \"maas-api-revoke-keys-e2e-autores-c1e164-d5g9d\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.587482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.587463 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c1e164-d5g9d\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.689030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.688990 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwz27\" (UniqueName: \"kubernetes.io/projected/4df618e7-36a4-4859-b21b-0400e64d5f2c-kube-api-access-rwz27\") pod \"maas-api-revoke-keys-e2e-autores-c1e164-d5g9d\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.689219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.689054 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c1e164-d5g9d\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.689219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.689093 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c1e164-d5g9d\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.689841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.689817 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c1e164-d5g9d\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.691545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.691525 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c1e164-d5g9d\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.710134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.710107 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwz27\" (UniqueName: \"kubernetes.io/projected/4df618e7-36a4-4859-b21b-0400e64d5f2c-kube-api-access-rwz27\") pod \"maas-api-revoke-keys-e2e-autores-c1e164-d5g9d\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:53.822636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:53.822540 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:54.127652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:54.127600 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d"] Oct 29 23:24:54.130388 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:24:54.130357 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df618e7_36a4_4859_b21b_0400e64d5f2c.slice/crio-4bc28f405a10f6a3226d9e96a374db124b419046b4c7b93ac82e40543461a2af WatchSource:0}: Error finding container 4bc28f405a10f6a3226d9e96a374db124b419046b4c7b93ac82e40543461a2af: Status 404 returned error can't find the container with id 4bc28f405a10f6a3226d9e96a374db124b419046b4c7b93ac82e40543461a2af Oct 29 23:24:54.377574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:54.377484 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" event={"ID":"4df618e7-36a4-4859-b21b-0400e64d5f2c","Type":"ContainerStarted","Data":"4bc28f405a10f6a3226d9e96a374db124b419046b4c7b93ac82e40543461a2af"} Oct 29 23:24:55.388199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:55.388111 5736 generic.go:356] "Generic (PLEG): container finished" podID="4df618e7-36a4-4859-b21b-0400e64d5f2c" containerID="9542e3f5e7674decbbd5a8552bacd1086d906c50dccf7bc2402154d51236669e" exitCode=0 Oct 29 23:24:55.388199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:55.388155 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" event={"ID":"4df618e7-36a4-4859-b21b-0400e64d5f2c","Type":"ContainerDied","Data":"9542e3f5e7674decbbd5a8552bacd1086d906c50dccf7bc2402154d51236669e"} Oct 29 23:24:56.571222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.571196 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:56.642143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.642110 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-openshift-ca\") pod \"4df618e7-36a4-4859-b21b-0400e64d5f2c\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " Oct 29 23:24:56.642338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.642313 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df618e7-36a4-4859-b21b-0400e64d5f2c-kube-api-access-rwz27\" (UniqueName: \"kubernetes.io/projected/4df618e7-36a4-4859-b21b-0400e64d5f2c-kube-api-access-rwz27\") pod \"4df618e7-36a4-4859-b21b-0400e64d5f2c\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " Oct 29 23:24:56.642455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.642439 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-xks-ca\") pod \"4df618e7-36a4-4859-b21b-0400e64d5f2c\" (UID: \"4df618e7-36a4-4859-b21b-0400e64d5f2c\") " Oct 29 23:24:56.642622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.642594 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-openshift-ca" pod "4df618e7-36a4-4859-b21b-0400e64d5f2c" (UID: "4df618e7-36a4-4859-b21b-0400e64d5f2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:24:56.643182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.643155 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:24:56.649766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.649703 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df618e7-36a4-4859-b21b-0400e64d5f2c-kube-api-access-rwz27" pod "4df618e7-36a4-4859-b21b-0400e64d5f2c" (UID: "4df618e7-36a4-4859-b21b-0400e64d5f2c"). InnerVolumeSpecName "kube-api-access-rwz27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:56.650103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.650078 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-xks-ca" pod "4df618e7-36a4-4859-b21b-0400e64d5f2c" (UID: "4df618e7-36a4-4859-b21b-0400e64d5f2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:24:56.744657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.744629 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwz27\" (UniqueName: \"kubernetes.io/projected/4df618e7-36a4-4859-b21b-0400e64d5f2c-kube-api-access-rwz27\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:24:56.744657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:56.744658 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df618e7-36a4-4859-b21b-0400e64d5f2c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:24:56.826235 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:24:56.826191 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:24:57.410502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:57.410429 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" Oct 29 23:24:57.411318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:57.410574 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d" event={"ID":"4df618e7-36a4-4859-b21b-0400e64d5f2c","Type":"ContainerDied","Data":"4bc28f405a10f6a3226d9e96a374db124b419046b4c7b93ac82e40543461a2af"} Oct 29 23:24:57.411514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:24:57.411499 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bc28f405a10f6a3226d9e96a374db124b419046b4c7b93ac82e40543461a2af" Oct 29 23:25:08.173284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.173245 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" podUID="38d9982f-cecd-44fe-ae74-a8172a361338" containerName="main" containerID="cri-o://f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd" gracePeriod=45 Oct 29 23:25:08.481301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.481272 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:25:08.499038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.499011 5736 generic.go:356] "Generic (PLEG): container finished" podID="38d9982f-cecd-44fe-ae74-a8172a361338" containerID="f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd" exitCode=0 Oct 29 23:25:08.499175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.499050 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" Oct 29 23:25:08.499175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.499095 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" event={"ID":"38d9982f-cecd-44fe-ae74-a8172a361338","Type":"ContainerDied","Data":"f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd"} Oct 29 23:25:08.499175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.499149 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz" event={"ID":"38d9982f-cecd-44fe-ae74-a8172a361338","Type":"ContainerDied","Data":"17f3c027d306fbda70ba8e8cba4fa9461dd517cee162fa70d8482ce476817979"} Oct 29 23:25:08.499175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.499170 5736 scope.go:122] "RemoveContainer" containerID="f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd" Oct 29 23:25:08.509546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.509527 5736 scope.go:122] "RemoveContainer" containerID="f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd" Oct 29 23:25:08.509916 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:25:08.509881 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd\": container with ID starting with f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd not found: ID does not exist" containerID="f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd" Oct 29 23:25:08.510018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.509921 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd"} err="failed to get container status \"f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd\": rpc error: code = NotFound desc = could not find container \"f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd\": container with ID starting with f8e1dc25dbf4d5a231d0c85fa2eb7e2930f45df67c9360742cec703fb41913dd not found: ID does not exist" Oct 29 23:25:08.637922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.637887 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-tmp-dir\") pod \"38d9982f-cecd-44fe-ae74-a8172a361338\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " Oct 29 23:25:08.638116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.638045 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-model-cache\") pod \"38d9982f-cecd-44fe-ae74-a8172a361338\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " Oct 29 23:25:08.638116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.638099 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-dshm\") pod \"38d9982f-cecd-44fe-ae74-a8172a361338\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " Oct 29 23:25:08.638258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.638161 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38d9982f-cecd-44fe-ae74-a8172a361338-tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9982f-cecd-44fe-ae74-a8172a361338-tls-certs\") pod \"38d9982f-cecd-44fe-ae74-a8172a361338\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " Oct 29 23:25:08.638258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.638185 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d9982f-cecd-44fe-ae74-a8172a361338-kube-api-access-rfl77\" (UniqueName: \"kubernetes.io/projected/38d9982f-cecd-44fe-ae74-a8172a361338-kube-api-access-rfl77\") pod \"38d9982f-cecd-44fe-ae74-a8172a361338\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " Oct 29 23:25:08.638258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.638201 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-tmp-dir" pod "38d9982f-cecd-44fe-ae74-a8172a361338" (UID: "38d9982f-cecd-44fe-ae74-a8172a361338"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:08.638434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.638303 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-home\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-home\") pod \"38d9982f-cecd-44fe-ae74-a8172a361338\" (UID: \"38d9982f-cecd-44fe-ae74-a8172a361338\") " Oct 29 23:25:08.638434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.638315 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-model-cache" pod "38d9982f-cecd-44fe-ae74-a8172a361338" (UID: "38d9982f-cecd-44fe-ae74-a8172a361338"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:08.638574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.638554 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-home" pod "38d9982f-cecd-44fe-ae74-a8172a361338" (UID: "38d9982f-cecd-44fe-ae74-a8172a361338"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:08.639019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.638999 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:08.639112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.639025 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:08.639112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.639040 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:08.640509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.640483 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d9982f-cecd-44fe-ae74-a8172a361338-tls-certs" pod "38d9982f-cecd-44fe-ae74-a8172a361338" (UID: "38d9982f-cecd-44fe-ae74-a8172a361338"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:08.640884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.640858 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d9982f-cecd-44fe-ae74-a8172a361338-kube-api-access-rfl77" pod "38d9982f-cecd-44fe-ae74-a8172a361338" (UID: "38d9982f-cecd-44fe-ae74-a8172a361338"). InnerVolumeSpecName "kube-api-access-rfl77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:08.640965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.640881 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-dshm" pod "38d9982f-cecd-44fe-ae74-a8172a361338" (UID: "38d9982f-cecd-44fe-ae74-a8172a361338"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:08.740143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.740111 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9982f-cecd-44fe-ae74-a8172a361338-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:08.740143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.740141 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9982f-cecd-44fe-ae74-a8172a361338-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:08.740332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.740152 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfl77\" (UniqueName: \"kubernetes.io/projected/38d9982f-cecd-44fe-ae74-a8172a361338-kube-api-access-rfl77\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:08.842960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.842928 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz"] Oct 29 23:25:08.852078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:08.852051 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c1e164/autores-model-c1e164-kserve-75fc747c97-4mbhz"] Oct 29 23:25:09.005766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:09.005689 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d9982f-cecd-44fe-ae74-a8172a361338" path="/var/lib/kubelet/pods/38d9982f-cecd-44fe-ae74-a8172a361338/volumes" Oct 29 23:25:10.028187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:10.028147 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d"] Oct 29 23:25:10.036945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:10.036909 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c1e164-d5g9d"] Oct 29 23:25:11.005483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:11.005448 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df618e7-36a4-4859-b21b-0400e64d5f2c" path="/var/lib/kubelet/pods/4df618e7-36a4-4859-b21b-0400e64d5f2c/volumes" Oct 29 23:25:22.586597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.586546 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m"] Oct 29 23:25:22.587250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.587231 5736 cpu_manager.go:414] "containerMap: removing container" podUID="38d9982f-cecd-44fe-ae74-a8172a361338" containerName="main" Oct 29 23:25:22.587250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.587249 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d9982f-cecd-44fe-ae74-a8172a361338" containerName="main" Oct 29 23:25:22.587414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.587287 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4df618e7-36a4-4859-b21b-0400e64d5f2c" containerName="revoke-keys" Oct 29 23:25:22.587414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.587295 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df618e7-36a4-4859-b21b-0400e64d5f2c" containerName="revoke-keys" Oct 29 23:25:22.587538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.587416 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d9982f-cecd-44fe-ae74-a8172a361338" containerName="main" Oct 29 23:25:22.587538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.587432 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df618e7-36a4-4859-b21b-0400e64d5f2c" containerName="revoke-keys" Oct 29 23:25:22.591213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.591181 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.603906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.603886 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:25:22.614044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.614000 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:25:22.618211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.618183 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m"] Oct 29 23:25:22.749212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.749163 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c767c33f-zxz5m\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.749488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.749464 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c767c33f-zxz5m\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.749555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.749498 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkl8\" (UniqueName: \"kubernetes.io/projected/4e184156-1c95-4871-9104-7a7a7151f73c-kube-api-access-7kkl8\") pod \"maas-api-revoke-sub-c767c33f-zxz5m\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.850701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.850572 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c767c33f-zxz5m\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.850701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.850659 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkl8\" (UniqueName: \"kubernetes.io/projected/4e184156-1c95-4871-9104-7a7a7151f73c-kube-api-access-7kkl8\") pod \"maas-api-revoke-sub-c767c33f-zxz5m\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.850948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.850740 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c767c33f-zxz5m\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.851400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.851378 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c767c33f-zxz5m\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.852950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.852928 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c767c33f-zxz5m\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.872388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.872362 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkl8\" (UniqueName: \"kubernetes.io/projected/4e184156-1c95-4871-9104-7a7a7151f73c-kube-api-access-7kkl8\") pod \"maas-api-revoke-sub-c767c33f-zxz5m\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:22.905817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:22.905790 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:23.254533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:23.253409 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m"] Oct 29 23:25:23.266195 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:25:23.266157 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e184156_1c95_4871_9104_7a7a7151f73c.slice/crio-ba42c1cdab077d2969513a468952b88e7b68c781941af5a1270aae765e619b15 WatchSource:0}: Error finding container ba42c1cdab077d2969513a468952b88e7b68c781941af5a1270aae765e619b15: Status 404 returned error can't find the container with id ba42c1cdab077d2969513a468952b88e7b68c781941af5a1270aae765e619b15 Oct 29 23:25:23.749753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:23.749715 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" event={"ID":"4e184156-1c95-4871-9104-7a7a7151f73c","Type":"ContainerStarted","Data":"84f6a1d5d16ad51fe0ebbb7e587cfe01840bc0287019e8e87eb44ab8415ac79e"} Oct 29 23:25:23.750154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:23.749771 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" event={"ID":"4e184156-1c95-4871-9104-7a7a7151f73c","Type":"ContainerStarted","Data":"ba42c1cdab077d2969513a468952b88e7b68c781941af5a1270aae765e619b15"} Oct 29 23:25:23.786382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:23.786332 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" podStartSLOduration=1.786320382 podStartE2EDuration="1.786320382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:23.785199353 +0000 UTC m=+2520.901118960" watchObservedRunningTime="2026-10-29 23:25:23.786320382 +0000 UTC m=+2520.902239988" Oct 29 23:25:24.759233 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:24.759192 5736 generic.go:356] "Generic (PLEG): container finished" podID="4e184156-1c95-4871-9104-7a7a7151f73c" containerID="84f6a1d5d16ad51fe0ebbb7e587cfe01840bc0287019e8e87eb44ab8415ac79e" exitCode=0 Oct 29 23:25:24.759655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:24.759268 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" event={"ID":"4e184156-1c95-4871-9104-7a7a7151f73c","Type":"ContainerDied","Data":"84f6a1d5d16ad51fe0ebbb7e587cfe01840bc0287019e8e87eb44ab8415ac79e"} Oct 29 23:25:25.920054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:25.920024 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:25.994743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:25.994712 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-xks-ca\") pod \"4e184156-1c95-4871-9104-7a7a7151f73c\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " Oct 29 23:25:25.994900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:25.994849 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e184156-1c95-4871-9104-7a7a7151f73c-kube-api-access-7kkl8\" (UniqueName: \"kubernetes.io/projected/4e184156-1c95-4871-9104-7a7a7151f73c-kube-api-access-7kkl8\") pod \"4e184156-1c95-4871-9104-7a7a7151f73c\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " Oct 29 23:25:25.994900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:25.994897 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-openshift-ca\") pod \"4e184156-1c95-4871-9104-7a7a7151f73c\" (UID: \"4e184156-1c95-4871-9104-7a7a7151f73c\") " Oct 29 23:25:25.995307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:25.995281 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-openshift-ca" pod "4e184156-1c95-4871-9104-7a7a7151f73c" (UID: "4e184156-1c95-4871-9104-7a7a7151f73c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:25.995515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:25.995498 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:25.997003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:25.996979 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-xks-ca" pod "4e184156-1c95-4871-9104-7a7a7151f73c" (UID: "4e184156-1c95-4871-9104-7a7a7151f73c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:25.997421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:25.997396 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e184156-1c95-4871-9104-7a7a7151f73c-kube-api-access-7kkl8" pod "4e184156-1c95-4871-9104-7a7a7151f73c" (UID: "4e184156-1c95-4871-9104-7a7a7151f73c"). InnerVolumeSpecName "kube-api-access-7kkl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:26.096544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:26.096473 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e184156-1c95-4871-9104-7a7a7151f73c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:26.096544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:26.096507 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kkl8\" (UniqueName: \"kubernetes.io/projected/4e184156-1c95-4871-9104-7a7a7151f73c-kube-api-access-7kkl8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:26.777123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:26.777091 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" Oct 29 23:25:26.777123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:26.777114 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m" event={"ID":"4e184156-1c95-4871-9104-7a7a7151f73c","Type":"ContainerDied","Data":"ba42c1cdab077d2969513a468952b88e7b68c781941af5a1270aae765e619b15"} Oct 29 23:25:26.777345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:26.777170 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba42c1cdab077d2969513a468952b88e7b68c781941af5a1270aae765e619b15" Oct 29 23:25:26.825956 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:25:26.825919 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:25:26.923740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:26.923709 5736 scope.go:122] "RemoveContainer" containerID="eeb1abe12fa3e31af2392681cb79f4eb545d2c9bf5bf627832ed727e21db233a" Oct 29 23:25:33.996806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:33.996717 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5"] Oct 29 23:25:38.997327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:38.997261 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" podUID="55be6c9f-0e46-47ce-84a0-0c2dab8bd577" containerName="payload-processing" containerID="cri-o://13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351" gracePeriod=25 Oct 29 23:25:39.313896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.313859 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:25:39.404341 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.404288 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-provider-credentials\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-provider-credentials\") pod \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " Oct 29 23:25:39.404540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.404409 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-routing-overlay-volume\") pod \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " Oct 29 23:25:39.404540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.404476 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-plugins-config-volume\") pod \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " Oct 29 23:25:39.404731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.404715 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-kube-api-access-xl6kg\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-kube-api-access-xl6kg\") pod \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\" (UID: \"55be6c9f-0e46-47ce-84a0-0c2dab8bd577\") " Oct 29 23:25:39.405192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.405166 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-plugins-config-volume" pod "55be6c9f-0e46-47ce-84a0-0c2dab8bd577" (UID: "55be6c9f-0e46-47ce-84a0-0c2dab8bd577"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:39.409778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.409747 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-kube-api-access-xl6kg" pod "55be6c9f-0e46-47ce-84a0-0c2dab8bd577" (UID: "55be6c9f-0e46-47ce-84a0-0c2dab8bd577"). InnerVolumeSpecName "kube-api-access-xl6kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:39.409937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.409815 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-provider-credentials" pod "55be6c9f-0e46-47ce-84a0-0c2dab8bd577" (UID: "55be6c9f-0e46-47ce-84a0-0c2dab8bd577"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:39.412387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.412356 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-routing-overlay-volume" pod "55be6c9f-0e46-47ce-84a0-0c2dab8bd577" (UID: "55be6c9f-0e46-47ce-84a0-0c2dab8bd577"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:39.506405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.506320 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:39.506405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.506358 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:39.506405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.506373 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6kg\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-kube-api-access-xl6kg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:39.506405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.506386 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55be6c9f-0e46-47ce-84a0-0c2dab8bd577-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:25:39.880955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.880865 5736 generic.go:356] "Generic (PLEG): container finished" podID="55be6c9f-0e46-47ce-84a0-0c2dab8bd577" containerID="13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351" exitCode=0 Oct 29 23:25:39.880955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.880915 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" Oct 29 23:25:39.881160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.880950 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" event={"ID":"55be6c9f-0e46-47ce-84a0-0c2dab8bd577","Type":"ContainerDied","Data":"13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351"} Oct 29 23:25:39.881160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.881008 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5" event={"ID":"55be6c9f-0e46-47ce-84a0-0c2dab8bd577","Type":"ContainerDied","Data":"3b17069cc55ed34fd2c03c9c126ef7a15600927c9be76268a5f3bc88a6f3014f"} Oct 29 23:25:39.881160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.881030 5736 scope.go:122] "RemoveContainer" containerID="13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351" Oct 29 23:25:39.893306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.893284 5736 scope.go:122] "RemoveContainer" containerID="13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351" Oct 29 23:25:39.893624 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:25:39.893555 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351\": container with ID starting with 13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351 not found: ID does not exist" containerID="13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351" Oct 29 23:25:39.893722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.893638 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351"} err="failed to get container status \"13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351\": rpc error: code = NotFound desc = could not find container \"13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351\": container with ID starting with 13bafb58a209641fc934afd8dd7bc7e4618646166da40d6d05c9250e8afb4351 not found: ID does not exist" Oct 29 23:25:39.928312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.928287 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5"] Oct 29 23:25:39.938008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:39.937986 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-646bqhmw5"] Oct 29 23:25:41.004208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:41.004173 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55be6c9f-0e46-47ce-84a0-0c2dab8bd577" path="/var/lib/kubelet/pods/55be6c9f-0e46-47ce-84a0-0c2dab8bd577/volumes" Oct 29 23:25:48.067567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.067529 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx"] Oct 29 23:25:48.068237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.068218 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4e184156-1c95-4871-9104-7a7a7151f73c" containerName="revoke-keys" Oct 29 23:25:48.068290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.068239 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e184156-1c95-4871-9104-7a7a7151f73c" containerName="revoke-keys" Oct 29 23:25:48.068328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.068297 5736 cpu_manager.go:414] "containerMap: removing container" podUID="55be6c9f-0e46-47ce-84a0-0c2dab8bd577" containerName="payload-processing" Oct 29 23:25:48.068328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.068304 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55be6c9f-0e46-47ce-84a0-0c2dab8bd577" containerName="payload-processing" Oct 29 23:25:48.068420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.068410 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="55be6c9f-0e46-47ce-84a0-0c2dab8bd577" containerName="payload-processing" Oct 29 23:25:48.068463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.068423 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e184156-1c95-4871-9104-7a7a7151f73c" containerName="revoke-keys" Oct 29 23:25:48.073853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.073831 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.091566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.091533 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx"] Oct 29 23:25:48.097299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.097257 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"openshift-service-ca.crt\"" Oct 29 23:25:48.097299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.097278 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"kube-root-ca.crt\"" Oct 29 23:25:48.097481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.097278 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"agc-cred-68806e843198\"" Oct 29 23:25:48.097481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.097256 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4e2ac7\"" Oct 29 23:25:48.097481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.097341 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"routing-overlay\"" Oct 29 23:25:48.099386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.099368 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"payload-processing-external-model-e2e--a44b4792-dockercfg-rqjzx\"" Oct 29 23:25:48.200601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.193394 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.200601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.193479 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhckr\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-kube-api-access-lhckr\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.200601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.193542 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.200601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.193600 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.305606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.305544 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.305831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.305645 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.316265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.306645 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.316265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.306760 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.316265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.306818 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhckr\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-kube-api-access-lhckr\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.316265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.309558 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.344146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.339446 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhckr\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-kube-api-access-lhckr\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.344146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.339987 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.388703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.388663 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:48.851259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.851223 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx"] Oct 29 23:25:48.984638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:48.984606 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" event={"ID":"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e","Type":"ContainerStarted","Data":"97d41a32efb177880be53f0dd61947c48a4f851a80294bc35ac5e4ab5467b0ea"} Oct 29 23:25:49.996935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:49.996895 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" event={"ID":"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e","Type":"ContainerStarted","Data":"24d945b0f6fd7cbb98351c80660e1cff0ec1dde0cc2293b36461103b6606895a"} Oct 29 23:25:49.997718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:25:49.997696 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:25:56.826303 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:25:56.826265 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:26:02.017184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:02.017144 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:26:02.058519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:02.058464 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" podStartSLOduration=14.058450415 podStartE2EDuration="14.058450415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:50.034265096 +0000 UTC m=+2547.150184703" watchObservedRunningTime="2026-10-29 23:26:02.058450415 +0000 UTC m=+2559.174370066" Oct 29 23:26:19.588645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.588604 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd"] Oct 29 23:26:19.594190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.594163 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.612837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.612810 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd"] Oct 29 23:26:19.617290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.617265 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dea9cb\"/\"agc-cred-189f55b8d790\"" Oct 29 23:26:19.719768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.719733 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.719955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.719866 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.719955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.719915 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-kube-api-access-wxd27\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.720049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.720035 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.821788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.821746 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.822134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.822110 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.822293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.822275 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-kube-api-access-wxd27\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.822518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.822501 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.828598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.823424 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.828804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.823499 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.877649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.859849 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-kube-api-access-wxd27\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.897302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.897263 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:19.917653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:19.917624 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:20.429649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.426514 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd"] Oct 29 23:26:20.451688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.451655 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs"] Oct 29 23:26:20.455972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.455939 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.465957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.465930 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd"] Oct 29 23:26:20.468209 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:26:20.468175 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf538f70d_3ad2_4871_8a0b_f3705d4b9c54.slice/crio-9bbee1d99b241b0e09fb018cf6a04602d1f7d7004e89b4044469c2212879408e WatchSource:0}: Error finding container 9bbee1d99b241b0e09fb018cf6a04602d1f7d7004e89b4044469c2212879408e: Status 404 returned error can't find the container with id 9bbee1d99b241b0e09fb018cf6a04602d1f7d7004e89b4044469c2212879408e Oct 29 23:26:20.471882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.471865 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dea9cb\"/\"agc-cred-973f5d3fe437\"" Oct 29 23:26:20.476789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.476764 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs"] Oct 29 23:26:20.563941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.563906 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.564089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.564047 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.564164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.564095 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkgl5\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-kube-api-access-zkgl5\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.564348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.564250 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.665724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.665693 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.666234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.665799 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.666234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.665896 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.666234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.665936 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkgl5\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-kube-api-access-zkgl5\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.666678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.666654 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.666779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.666757 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.668530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.668508 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.687274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.687224 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkgl5\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-kube-api-access-zkgl5\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:20.791964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:20.791929 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:21.102194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:21.102167 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs"] Oct 29 23:26:21.104147 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:26:21.104117 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f7b37e_e93a_4ab5_8f18_bd194b8f966d.slice/crio-6d82f1c24b6e44ee6f6898c7697c57b67a0623f84939021edcc1bdc45fdf4ef4 WatchSource:0}: Error finding container 6d82f1c24b6e44ee6f6898c7697c57b67a0623f84939021edcc1bdc45fdf4ef4: Status 404 returned error can't find the container with id 6d82f1c24b6e44ee6f6898c7697c57b67a0623f84939021edcc1bdc45fdf4ef4 Oct 29 23:26:21.353850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:21.353761 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" event={"ID":"f538f70d-3ad2-4871-8a0b-f3705d4b9c54","Type":"ContainerStarted","Data":"76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22"} Oct 29 23:26:21.353850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:21.353830 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:21.354063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:21.353858 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" event={"ID":"f538f70d-3ad2-4871-8a0b-f3705d4b9c54","Type":"ContainerStarted","Data":"9bbee1d99b241b0e09fb018cf6a04602d1f7d7004e89b4044469c2212879408e"} Oct 29 23:26:21.355512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:21.355477 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" event={"ID":"58f7b37e-e93a-4ab5-8f18-bd194b8f966d","Type":"ContainerStarted","Data":"342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326"} Oct 29 23:26:21.355704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:21.355528 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" event={"ID":"58f7b37e-e93a-4ab5-8f18-bd194b8f966d","Type":"ContainerStarted","Data":"6d82f1c24b6e44ee6f6898c7697c57b67a0623f84939021edcc1bdc45fdf4ef4"} Oct 29 23:26:21.355704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:21.355556 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:21.391998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:21.391947 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" podStartSLOduration=2.3919329879999998 podStartE2EDuration="2.391932988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:21.390648553 +0000 UTC m=+2578.506568156" watchObservedRunningTime="2026-10-29 23:26:21.391932988 +0000 UTC m=+2578.507852593" Oct 29 23:26:21.431429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:21.431376 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" podStartSLOduration=1.431358615 podStartE2EDuration="1.431358615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:21.429283686 +0000 UTC m=+2578.545203304" watchObservedRunningTime="2026-10-29 23:26:21.431358615 +0000 UTC m=+2578.547278225" Oct 29 23:26:24.030352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.030313 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr"] Oct 29 23:26:24.040102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.040078 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6b35a3-ttqlr"] Oct 29 23:26:24.333467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.333379 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r"] Oct 29 23:26:24.338866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.338844 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.363660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.363635 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r"] Oct 29 23:26:24.426826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.426791 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.426994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.426843 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.426994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.426972 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.427070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.427012 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.427070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.427037 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdaaaba-4830-4237-9850-094ec264d63d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.427160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.427144 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.427221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.427208 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.427258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.427238 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf8tw\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-kube-api-access-hf8tw\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.427321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.427308 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdaaaba-4830-4237-9850-094ec264d63d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528378 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528424 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528476 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528500 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528515 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdaaaba-4830-4237-9850-094ec264d63d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528659 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528707 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528745 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf8tw\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-kube-api-access-hf8tw\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528804 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdaaaba-4830-4237-9850-094ec264d63d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.528959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.528867 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.529211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.529041 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.529211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.529074 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.529316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.529279 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.529550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.529532 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdaaaba-4830-4237-9850-094ec264d63d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.530806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.530784 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.531119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.531100 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdaaaba-4830-4237-9850-094ec264d63d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.549420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.549398 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.549748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.549731 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf8tw\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-kube-api-access-hf8tw\") pod \"maas-default-gateway-openshift-default-55fbd65575-rz49r\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.653095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.653027 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:24.996058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:24.996032 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r"] Oct 29 23:26:24.998297 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:26:24.998268 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdaaaba_4830_4237_9850_094ec264d63d.slice/crio-b97906f51447f98fa9a912d1ac551e476d3f34e11bc40170fbb1d8c9bb84e186 WatchSource:0}: Error finding container b97906f51447f98fa9a912d1ac551e476d3f34e11bc40170fbb1d8c9bb84e186: Status 404 returned error can't find the container with id b97906f51447f98fa9a912d1ac551e476d3f34e11bc40170fbb1d8c9bb84e186 Oct 29 23:26:25.001463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:25.001425 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:26:25.001631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:25.001509 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:26:25.001631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:25.001556 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:26:25.005801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:25.005776 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ec3f89-7376-4f75-a117-cd175ede6ccb" path="/var/lib/kubelet/pods/98ec3f89-7376-4f75-a117-cd175ede6ccb/volumes" Oct 29 23:26:25.406342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:25.406242 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" event={"ID":"ccdaaaba-4830-4237-9850-094ec264d63d","Type":"ContainerStarted","Data":"26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914"} Oct 29 23:26:25.406342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:25.406312 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" event={"ID":"ccdaaaba-4830-4237-9850-094ec264d63d","Type":"ContainerStarted","Data":"b97906f51447f98fa9a912d1ac551e476d3f34e11bc40170fbb1d8c9bb84e186"} Oct 29 23:26:25.448813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:25.448754 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" podStartSLOduration=1.448737946 podStartE2EDuration="1.448737946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:25.448128493 +0000 UTC m=+2582.564048100" watchObservedRunningTime="2026-10-29 23:26:25.448737946 +0000 UTC m=+2582.564657556" Oct 29 23:26:25.654255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:25.654212 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:26.355705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.355640 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" podUID="f538f70d-3ad2-4871-8a0b-f3705d4b9c54" containerName="payload-processing" containerID="cri-o://76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22" gracePeriod=25 Oct 29 23:26:26.659447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.659405 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-rz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:26:26.659875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.659474 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 29 23:26:26.679379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.679350 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:26.686719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.686695 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-plugins-config-volume\") pod \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " Oct 29 23:26:26.686850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.686765 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-kube-api-access-wxd27\") pod \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " Oct 29 23:26:26.686923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.686881 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-provider-credentials\") pod \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " Oct 29 23:26:26.686995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.686948 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-routing-overlay-volume\") pod \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\" (UID: \"f538f70d-3ad2-4871-8a0b-f3705d4b9c54\") " Oct 29 23:26:26.687145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.687119 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-plugins-config-volume" pod "f538f70d-3ad2-4871-8a0b-f3705d4b9c54" (UID: "f538f70d-3ad2-4871-8a0b-f3705d4b9c54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:26.687325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.687303 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-routing-overlay-volume" pod "f538f70d-3ad2-4871-8a0b-f3705d4b9c54" (UID: "f538f70d-3ad2-4871-8a0b-f3705d4b9c54"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:26.687603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.687571 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:26:26.687663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.687608 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:26:26.689065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.689043 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-kube-api-access-wxd27" pod "f538f70d-3ad2-4871-8a0b-f3705d4b9c54" (UID: "f538f70d-3ad2-4871-8a0b-f3705d4b9c54"). InnerVolumeSpecName "kube-api-access-wxd27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:26.689169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.689094 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-provider-credentials" pod "f538f70d-3ad2-4871-8a0b-f3705d4b9c54" (UID: "f538f70d-3ad2-4871-8a0b-f3705d4b9c54"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:26.789035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.788995 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-kube-api-access-wxd27\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:26:26.789035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.789036 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f538f70d-3ad2-4871-8a0b-f3705d4b9c54-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:26:26.825381 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:26:26.825339 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:26:26.959310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:26.959287 5736 scope.go:122] "RemoveContainer" containerID="0f490b320ba761f33e205ae0899134e17bc93ffc0cb8b708b248283fe5da59f4" Oct 29 23:26:27.427024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.426938 5736 generic.go:356] "Generic (PLEG): container finished" podID="f538f70d-3ad2-4871-8a0b-f3705d4b9c54" containerID="76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22" exitCode=0 Oct 29 23:26:27.427024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.426984 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" Oct 29 23:26:27.427228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.427026 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" event={"ID":"f538f70d-3ad2-4871-8a0b-f3705d4b9c54","Type":"ContainerDied","Data":"76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22"} Oct 29 23:26:27.427228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.427083 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd" event={"ID":"f538f70d-3ad2-4871-8a0b-f3705d4b9c54","Type":"ContainerDied","Data":"9bbee1d99b241b0e09fb018cf6a04602d1f7d7004e89b4044469c2212879408e"} Oct 29 23:26:27.427228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.427112 5736 scope.go:122] "RemoveContainer" containerID="76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22" Oct 29 23:26:27.463338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.463317 5736 scope.go:122] "RemoveContainer" containerID="76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22" Oct 29 23:26:27.463660 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:26:27.463639 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22\": container with ID starting with 76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22 not found: ID does not exist" containerID="76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22" Oct 29 23:26:27.463753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.463668 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22"} err="failed to get container status \"76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22\": rpc error: code = NotFound desc = could not find container \"76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22\": container with ID starting with 76cfbdf408cb02acab8d1b2fa78d47063989d8e1e896b9748c26c1f91fe4da22 not found: ID does not exist" Oct 29 23:26:27.477973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.477946 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd"] Oct 29 23:26:27.487799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.487775 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc94hvxd"] Oct 29 23:26:27.654605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.653935 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-rz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:26:27.654605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:27.653991 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 29 23:26:28.654958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:28.654918 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-rz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:26:28.655471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:28.654984 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 29 23:26:29.007436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:29.007400 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f538f70d-3ad2-4871-8a0b-f3705d4b9c54" path="/var/lib/kubelet/pods/f538f70d-3ad2-4871-8a0b-f3705d4b9c54/volumes" Oct 29 23:26:29.654603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:29.654030 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-rz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:26:29.654603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:29.654120 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 29 23:26:30.670641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:30.667318 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-rz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:26:30.670641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:30.667380 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 29 23:26:31.653696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.653653 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-rz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:26:31.653899 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.653721 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 29 23:26:31.955138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.955047 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn"] Oct 29 23:26:31.956322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.955960 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f538f70d-3ad2-4871-8a0b-f3705d4b9c54" containerName="payload-processing" Oct 29 23:26:31.956322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.955985 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f538f70d-3ad2-4871-8a0b-f3705d4b9c54" containerName="payload-processing" Oct 29 23:26:31.956322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.956150 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f538f70d-3ad2-4871-8a0b-f3705d4b9c54" containerName="payload-processing" Oct 29 23:26:31.962124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.962101 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:31.983277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.983240 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn"] Oct 29 23:26:31.985300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.985263 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"kube-root-ca.crt\"" Oct 29 23:26:31.986145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.986121 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"openshift-service-ca.crt\"" Oct 29 23:26:31.986285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.986265 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fb367d\"" Oct 29 23:26:31.986893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:31.986873 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"routing-overlay\"" Oct 29 23:26:32.038935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.038900 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"agc-cred-4e0b6818b8ca\"" Oct 29 23:26:32.092765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.041766 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"payload-processing-external-model-e2e--06d8c548-dockercfg-bf6wn\"" Oct 29 23:26:32.092997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.077853 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.093422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.093394 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.093739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.093717 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr6b5\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-kube-api-access-qr6b5\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.093942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.093922 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.195444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.195390 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.196370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.196246 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.196944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.196920 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr6b5\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-kube-api-access-qr6b5\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.197160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.197136 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.198412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.197494 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.198789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.198368 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.228606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.223847 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.254494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.254459 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr6b5\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-kube-api-access-qr6b5\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.288641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.287407 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:32.371636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.369069 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:26:32.446564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.446533 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h"] Oct 29 23:26:32.654439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.653980 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-rz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Oct 29 23:26:32.654439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.654044 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Oct 29 23:26:32.678598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:32.678550 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn"] Oct 29 23:26:32.681242 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:26:32.681203 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe388bf_3ac9_40c1_9559_ba3b042df7c7.slice/crio-598fb027b2dd513be94110e14559286287e591373e24d1849940e3567916f750 WatchSource:0}: Error finding container 598fb027b2dd513be94110e14559286287e591373e24d1849940e3567916f750: Status 404 returned error can't find the container with id 598fb027b2dd513be94110e14559286287e591373e24d1849940e3567916f750 Oct 29 23:26:33.550400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:33.550362 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" event={"ID":"bfe388bf-3ac9-40c1-9559-ba3b042df7c7","Type":"ContainerStarted","Data":"2b04ddc58be1307a7b345e5ecb4dafe6b8d7190e6049a252eaf6037493b92329"} Oct 29 23:26:33.550863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:33.550423 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" event={"ID":"bfe388bf-3ac9-40c1-9559-ba3b042df7c7","Type":"ContainerStarted","Data":"598fb027b2dd513be94110e14559286287e591373e24d1849940e3567916f750"} Oct 29 23:26:33.550863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:33.550452 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:33.607921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:33.607839 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" podStartSLOduration=2.607819579 podStartE2EDuration="2.607819579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:33.588775409 +0000 UTC m=+2590.704695030" watchObservedRunningTime="2026-10-29 23:26:33.607819579 +0000 UTC m=+2590.723739186" Oct 29 23:26:34.242766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:34.242732 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:34.243263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:34.243214 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:34.244334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:34.244311 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:26:37.447623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.447552 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" podUID="bcd642cf-4946-445f-bd55-315a5703ba80" containerName="payload-processing" containerID="cri-o://1d11bcc53c4924944ef77ea375c05f24f4e60ad0cdff6e908516b8c05e2765a8" gracePeriod=25 Oct 29 23:26:37.588376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.588347 5736 generic.go:356] "Generic (PLEG): container finished" podID="bcd642cf-4946-445f-bd55-315a5703ba80" containerID="1d11bcc53c4924944ef77ea375c05f24f4e60ad0cdff6e908516b8c05e2765a8" exitCode=0 Oct 29 23:26:37.588512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.588426 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" event={"ID":"bcd642cf-4946-445f-bd55-315a5703ba80","Type":"ContainerDied","Data":"1d11bcc53c4924944ef77ea375c05f24f4e60ad0cdff6e908516b8c05e2765a8"} Oct 29 23:26:37.773068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.773039 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:26:37.916069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.916040 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-plugins-config-volume\") pod \"bcd642cf-4946-445f-bd55-315a5703ba80\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " Oct 29 23:26:37.916336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.916169 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-kube-api-access-f69lj\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-kube-api-access-f69lj\") pod \"bcd642cf-4946-445f-bd55-315a5703ba80\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " Oct 29 23:26:37.916336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.916222 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-provider-credentials\") pod \"bcd642cf-4946-445f-bd55-315a5703ba80\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " Oct 29 23:26:37.916336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.916277 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-routing-overlay-volume\") pod \"bcd642cf-4946-445f-bd55-315a5703ba80\" (UID: \"bcd642cf-4946-445f-bd55-315a5703ba80\") " Oct 29 23:26:37.916530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.916502 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-plugins-config-volume" pod "bcd642cf-4946-445f-bd55-315a5703ba80" (UID: "bcd642cf-4946-445f-bd55-315a5703ba80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:37.916933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.916754 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-routing-overlay-volume" pod "bcd642cf-4946-445f-bd55-315a5703ba80" (UID: "bcd642cf-4946-445f-bd55-315a5703ba80"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:37.920262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.920225 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:26:37.920262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.920262 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd642cf-4946-445f-bd55-315a5703ba80-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:26:37.926965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.926934 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-kube-api-access-f69lj" pod "bcd642cf-4946-445f-bd55-315a5703ba80" (UID: "bcd642cf-4946-445f-bd55-315a5703ba80"). InnerVolumeSpecName "kube-api-access-f69lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:37.930541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:37.930418 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-provider-credentials" pod "bcd642cf-4946-445f-bd55-315a5703ba80" (UID: "bcd642cf-4946-445f-bd55-315a5703ba80"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:38.021651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:38.021575 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f69lj\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-kube-api-access-f69lj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:26:38.021651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:38.021614 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcd642cf-4946-445f-bd55-315a5703ba80-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:26:38.598401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:38.598367 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" Oct 29 23:26:38.598401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:38.598385 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h" event={"ID":"bcd642cf-4946-445f-bd55-315a5703ba80","Type":"ContainerDied","Data":"e40700c5f58588d3ed9e48107b672182b3854c7d04a8e9d1d7755812e3286a00"} Oct 29 23:26:38.598895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:38.598472 5736 scope.go:122] "RemoveContainer" containerID="1d11bcc53c4924944ef77ea375c05f24f4e60ad0cdff6e908516b8c05e2765a8" Oct 29 23:26:38.647652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:38.647620 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h"] Oct 29 23:26:38.657154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:38.657131 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47cft8h"] Oct 29 23:26:39.005385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:39.005354 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd642cf-4946-445f-bd55-315a5703ba80" path="/var/lib/kubelet/pods/bcd642cf-4946-445f-bd55-315a5703ba80/volumes" Oct 29 23:26:43.051606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:43.048860 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh"] Oct 29 23:26:43.060822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:43.060778 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6"] Oct 29 23:26:43.064270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:43.064242 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv"] Oct 29 23:26:43.067440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:43.067417 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2713814f-6jrw6"] Oct 29 23:26:43.070326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:43.070308 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81829674-bgnhh"] Oct 29 23:26:43.073179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:43.073155 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c8daed-qsfqv"] Oct 29 23:26:44.562553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:44.562522 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:26:45.005406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:45.005369 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad46e72-f065-4159-9270-28234a227d85" path="/var/lib/kubelet/pods/3ad46e72-f065-4159-9270-28234a227d85/volumes" Oct 29 23:26:45.005909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:45.005893 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a634441-1a2f-4e89-a3ec-169c5db13f15" path="/var/lib/kubelet/pods/6a634441-1a2f-4e89-a3ec-169c5db13f15/volumes" Oct 29 23:26:45.006335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:45.006322 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbcec93b-efd9-4733-a44d-34e9c2ff907d" path="/var/lib/kubelet/pods/bbcec93b-efd9-4733-a44d-34e9c2ff907d/volumes" Oct 29 23:26:46.039604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:46.030746 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx"] Oct 29 23:26:46.061384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:46.061349 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-275251cf-xd5kx"] Oct 29 23:26:47.005813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:47.005778 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f442c216-a87a-4e8f-baf8-73639be29834" path="/var/lib/kubelet/pods/f442c216-a87a-4e8f-baf8-73639be29834/volumes" Oct 29 23:26:51.743336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:51.743300 5736 generic.go:356] "Generic (PLEG): container finished" podID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerID="37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8" exitCode=0 Oct 29 23:26:51.743835 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:51.743365 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" event={"ID":"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac","Type":"ContainerDied","Data":"37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8"} Oct 29 23:26:51.743886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:51.743854 5736 scope.go:122] "RemoveContainer" containerID="37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8" Oct 29 23:26:51.745968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:51.745932 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:26:51.746075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:51.746006 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:26:51.746075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:51.746044 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:26:52.766475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:52.766438 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" event={"ID":"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac","Type":"ContainerStarted","Data":"b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487"} Oct 29 23:26:52.897238 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:52.897197 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:26:52.898827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:52.898802 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:26:52.898902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:52.898869 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 29 23:26:53.058277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:53.058167 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr"] Oct 29 23:26:53.062801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:53.062768 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss"] Oct 29 23:26:53.066303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:53.066281 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fce0643-m88wr"] Oct 29 23:26:53.069055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:53.069034 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd6930f-lvbss"] Oct 29 23:26:53.897287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:53.897247 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:26:53.897825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:53.897314 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 29 23:26:54.046663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:54.046626 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4"] Oct 29 23:26:54.057426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:54.057393 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8895b4-qzdm4"] Oct 29 23:26:54.901757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:54.901705 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:26:54.902288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:54.901810 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 29 23:26:54.976399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:54.976367 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:26:55.005396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:55.005368 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28962275-e0e4-420f-917f-c4e4115fa30a" path="/var/lib/kubelet/pods/28962275-e0e4-420f-917f-c4e4115fa30a/volumes" Oct 29 23:26:55.005883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:55.005868 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076" path="/var/lib/kubelet/pods/3d0a6fb0-fa9c-415a-8cc1-fd7abcbf1076/volumes" Oct 29 23:26:55.006297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:55.006284 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff" path="/var/lib/kubelet/pods/3d54fa65-aaf1-4479-8dd8-e3d24fd7d8ff/volumes" Oct 29 23:26:55.897709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:55.897668 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:26:55.897894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:55.897735 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 29 23:26:56.825447 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:26:56.825409 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:26:56.897625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:56.897561 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:26:56.897793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:56.897657 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 29 23:26:57.057892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:57.057853 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8"] Oct 29 23:26:57.061510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:57.061482 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccb0ef3-97df8"] Oct 29 23:26:57.897264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:57.897231 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 29 23:26:57.897705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:57.897281 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 29 23:26:58.902838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:58.902810 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:26:58.903986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:58.903962 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:26:59.006443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:26:59.006410 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acf648c5-e3af-42e9-a9a9-c885f79afaf5" path="/var/lib/kubelet/pods/acf648c5-e3af-42e9-a9a9-c885f79afaf5/volumes" Oct 29 23:27:18.991179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:18.991084 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t"] Oct 29 23:27:18.992515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:18.992484 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bcd642cf-4946-445f-bd55-315a5703ba80" containerName="payload-processing" Oct 29 23:27:18.992515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:18.992512 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd642cf-4946-445f-bd55-315a5703ba80" containerName="payload-processing" Oct 29 23:27:18.994790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:18.992780 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd642cf-4946-445f-bd55-315a5703ba80" containerName="payload-processing" Oct 29 23:27:18.998818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:18.998792 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.017734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.017704 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t"] Oct 29 23:27:19.021420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.021392 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10975\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e10975\"" Oct 29 23:27:19.021617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.021404 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10975\"/\"routing-overlay\"" Oct 29 23:27:19.021617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.021423 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10975\"/\"openshift-service-ca.crt\"" Oct 29 23:27:19.024404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.024364 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10975\"/\"e2e-cmp-legacy-b765fa-provider-cred\"" Oct 29 23:27:19.024530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.024436 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10975\"/\"kube-root-ca.crt\"" Oct 29 23:27:19.024530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.024436 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10975\"/\"payload-processing-external-model-e2e--625a6e10-dockercfg-jwztm\"" Oct 29 23:27:19.183133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.183094 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.183662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.183637 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.183986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.183963 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.184164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.184146 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbhz\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-kube-api-access-xvbhz\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.301646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.287704 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.301646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.287764 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbhz\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-kube-api-access-xvbhz\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.301646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.287929 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.301646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.288098 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.301646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.288921 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.301646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.289877 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.301646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.301525 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.326487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.326442 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbhz\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-kube-api-access-xvbhz\") pod \"payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.619481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.619406 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:19.931950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:19.931916 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t"] Oct 29 23:27:19.933190 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:27:19.933161 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d262f4a_f03e_4ba9_92cb_8cc3207181a7.slice/crio-c8dcec4ddd5cc24df2f762c2ed112cfddba6e4fda16f3cd3bea116bceb5d8083 WatchSource:0}: Error finding container c8dcec4ddd5cc24df2f762c2ed112cfddba6e4fda16f3cd3bea116bceb5d8083: Status 404 returned error can't find the container with id c8dcec4ddd5cc24df2f762c2ed112cfddba6e4fda16f3cd3bea116bceb5d8083 Oct 29 23:27:20.209127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:20.209038 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" event={"ID":"7d262f4a-f03e-4ba9-92cb-8cc3207181a7","Type":"ContainerStarted","Data":"ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546"} Oct 29 23:27:20.209127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:20.209119 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" event={"ID":"7d262f4a-f03e-4ba9-92cb-8cc3207181a7","Type":"ContainerStarted","Data":"c8dcec4ddd5cc24df2f762c2ed112cfddba6e4fda16f3cd3bea116bceb5d8083"} Oct 29 23:27:20.209572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:20.209181 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:20.250496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:20.250442 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" podStartSLOduration=2.250429256 podStartE2EDuration="2.250429256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:20.249928864 +0000 UTC m=+2637.365848483" watchObservedRunningTime="2026-10-29 23:27:20.250429256 +0000 UTC m=+2637.366348861" Oct 29 23:27:26.626671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.626639 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst"] Oct 29 23:27:26.631398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.631378 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.644558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.644536 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"e2e-cfl-817958-provider-cred\"" Oct 29 23:27:26.654879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.654852 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"payload-processing-external-model-e2e--ca6efe92-dockercfg-xw8pp\"" Oct 29 23:27:26.655023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.654938 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"kube-root-ca.crt\"" Oct 29 23:27:26.655113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.654944 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"routing-overlay\"" Oct 29 23:27:26.655193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.655065 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"openshift-service-ca.crt\"" Oct 29 23:27:26.655193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.655124 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c5ef2f\"" Oct 29 23:27:26.659227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.659203 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst"] Oct 29 23:27:26.677395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.677367 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.677527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.677440 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slhtm\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-kube-api-access-slhtm\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.677646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.677619 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.677759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.677742 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.779516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.779484 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.779717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.779690 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.779801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.779728 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slhtm\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-kube-api-access-slhtm\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.779801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.779788 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.780160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.780138 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.780405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.780385 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.782415 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.782394 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.800833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.800813 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slhtm\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-kube-api-access-slhtm\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:26.826694 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:27:26.826666 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:27:26.945770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:26.945691 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:27.055944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:27.055905 5736 scope.go:122] "RemoveContainer" containerID="6451a1105055f081c8fe0104ef1e88ffb6a4d116fbecb0bc8370917089470e08" Oct 29 23:27:27.258363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:27.258339 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst"] Oct 29 23:27:27.261140 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:27:27.261112 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa1e792_c7be_446e_850e_a645eb1f9df9.slice/crio-8956e6916117e607741d4e2aa24caf11422cd27a8f417798667dfc543749bcb8 WatchSource:0}: Error finding container 8956e6916117e607741d4e2aa24caf11422cd27a8f417798667dfc543749bcb8: Status 404 returned error can't find the container with id 8956e6916117e607741d4e2aa24caf11422cd27a8f417798667dfc543749bcb8 Oct 29 23:27:27.352260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:27.352230 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" event={"ID":"baa1e792-c7be-446e-850e-a645eb1f9df9","Type":"ContainerStarted","Data":"69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a"} Oct 29 23:27:27.352392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:27.352289 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" event={"ID":"baa1e792-c7be-446e-850e-a645eb1f9df9","Type":"ContainerStarted","Data":"8956e6916117e607741d4e2aa24caf11422cd27a8f417798667dfc543749bcb8"} Oct 29 23:27:27.352392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:27.352318 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:27.392706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:27.392497 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" podStartSLOduration=1.392481287 podStartE2EDuration="1.392481287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:27.389901452 +0000 UTC m=+2644.505821058" watchObservedRunningTime="2026-10-29 23:27:27.392481287 +0000 UTC m=+2644.508400894" Oct 29 23:27:30.282311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.282276 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8"] Oct 29 23:27:30.286987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.286967 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.311924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.311896 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:27:30.312056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.311970 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:27:30.316472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.316445 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8"] Oct 29 23:27:30.428493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.428455 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859b17bb-q6nf8\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.428701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.428520 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859b17bb-q6nf8\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.428701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.428680 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxd48\" (UniqueName: \"kubernetes.io/projected/33360daf-fb50-40fa-9e9f-67f16e31240f-kube-api-access-bxd48\") pod \"maas-api-revoke-sub-859b17bb-q6nf8\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.530310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.530281 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxd48\" (UniqueName: \"kubernetes.io/projected/33360daf-fb50-40fa-9e9f-67f16e31240f-kube-api-access-bxd48\") pod \"maas-api-revoke-sub-859b17bb-q6nf8\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.530461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.530442 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859b17bb-q6nf8\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.530505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.530495 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859b17bb-q6nf8\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.531248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.531224 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859b17bb-q6nf8\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.532894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.532843 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859b17bb-q6nf8\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.553662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.553637 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxd48\" (UniqueName: \"kubernetes.io/projected/33360daf-fb50-40fa-9e9f-67f16e31240f-kube-api-access-bxd48\") pod \"maas-api-revoke-sub-859b17bb-q6nf8\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.601656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.601637 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:30.916367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:30.916331 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8"] Oct 29 23:27:30.918778 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:27:30.918747 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33360daf_fb50_40fa_9e9f_67f16e31240f.slice/crio-148e2f9047028f4a340c174e313b333ad905f61b8fd80e9d9c89a8624b09ccc3 WatchSource:0}: Error finding container 148e2f9047028f4a340c174e313b333ad905f61b8fd80e9d9c89a8624b09ccc3: Status 404 returned error can't find the container with id 148e2f9047028f4a340c174e313b333ad905f61b8fd80e9d9c89a8624b09ccc3 Oct 29 23:27:31.219812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:31.219786 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:27:31.410622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:31.400133 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" event={"ID":"33360daf-fb50-40fa-9e9f-67f16e31240f","Type":"ContainerStarted","Data":"8079675c6e7f9b614ae62e52112cdd079db574a2dde0dffc79bf4d6b7ebe1dc2"} Oct 29 23:27:31.410622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:31.400252 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" event={"ID":"33360daf-fb50-40fa-9e9f-67f16e31240f","Type":"ContainerStarted","Data":"148e2f9047028f4a340c174e313b333ad905f61b8fd80e9d9c89a8624b09ccc3"} Oct 29 23:27:31.439363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:31.439300 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" podStartSLOduration=1.439286264 podStartE2EDuration="1.439286264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:31.437279953 +0000 UTC m=+2648.553199560" watchObservedRunningTime="2026-10-29 23:27:31.439286264 +0000 UTC m=+2648.555205869" Oct 29 23:27:32.416478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:32.416440 5736 generic.go:356] "Generic (PLEG): container finished" podID="33360daf-fb50-40fa-9e9f-67f16e31240f" containerID="8079675c6e7f9b614ae62e52112cdd079db574a2dde0dffc79bf4d6b7ebe1dc2" exitCode=0 Oct 29 23:27:32.416923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:32.416526 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" event={"ID":"33360daf-fb50-40fa-9e9f-67f16e31240f","Type":"ContainerDied","Data":"8079675c6e7f9b614ae62e52112cdd079db574a2dde0dffc79bf4d6b7ebe1dc2"} Oct 29 23:27:33.593115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.593084 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:33.673486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.673448 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33360daf-fb50-40fa-9e9f-67f16e31240f-kube-api-access-bxd48\" (UniqueName: \"kubernetes.io/projected/33360daf-fb50-40fa-9e9f-67f16e31240f-kube-api-access-bxd48\") pod \"33360daf-fb50-40fa-9e9f-67f16e31240f\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " Oct 29 23:27:33.673695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.673502 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-openshift-ca\") pod \"33360daf-fb50-40fa-9e9f-67f16e31240f\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " Oct 29 23:27:33.673779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.673705 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-xks-ca\") pod \"33360daf-fb50-40fa-9e9f-67f16e31240f\" (UID: \"33360daf-fb50-40fa-9e9f-67f16e31240f\") " Oct 29 23:27:33.674308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.674279 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-openshift-ca" pod "33360daf-fb50-40fa-9e9f-67f16e31240f" (UID: "33360daf-fb50-40fa-9e9f-67f16e31240f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:33.676075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.676050 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-xks-ca" pod "33360daf-fb50-40fa-9e9f-67f16e31240f" (UID: "33360daf-fb50-40fa-9e9f-67f16e31240f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:33.676223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.676206 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33360daf-fb50-40fa-9e9f-67f16e31240f-kube-api-access-bxd48" pod "33360daf-fb50-40fa-9e9f-67f16e31240f" (UID: "33360daf-fb50-40fa-9e9f-67f16e31240f"). InnerVolumeSpecName "kube-api-access-bxd48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:33.775958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.775930 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxd48\" (UniqueName: \"kubernetes.io/projected/33360daf-fb50-40fa-9e9f-67f16e31240f-kube-api-access-bxd48\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:27:33.775958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.775957 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:27:33.776114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:33.775967 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33360daf-fb50-40fa-9e9f-67f16e31240f-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:27:34.436369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:34.436339 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" event={"ID":"33360daf-fb50-40fa-9e9f-67f16e31240f","Type":"ContainerDied","Data":"148e2f9047028f4a340c174e313b333ad905f61b8fd80e9d9c89a8624b09ccc3"} Oct 29 23:27:34.436552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:34.436374 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8" Oct 29 23:27:34.436552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:34.436385 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="148e2f9047028f4a340c174e313b333ad905f61b8fd80e9d9c89a8624b09ccc3" Oct 29 23:27:38.364202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:38.364171 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:27:39.576656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.576620 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5"] Oct 29 23:27:39.577460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.577432 5736 cpu_manager.go:414] "containerMap: removing container" podUID="33360daf-fb50-40fa-9e9f-67f16e31240f" containerName="revoke-keys" Oct 29 23:27:39.577460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.577452 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33360daf-fb50-40fa-9e9f-67f16e31240f" containerName="revoke-keys" Oct 29 23:27:39.577643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.577623 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="33360daf-fb50-40fa-9e9f-67f16e31240f" containerName="revoke-keys" Oct 29 23:27:39.581739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.581718 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.605755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.605714 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.605907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.605779 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.605907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.605816 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.605907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.605853 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhswp\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-kube-api-access-lhswp\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.606078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.605906 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed043325-0e1b-4c86-89dc-fee8978ff145-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.606078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.605947 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.606078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.606002 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.606078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.606041 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.606283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.606107 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.608374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.608349 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5"] Oct 29 23:27:39.707951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.707917 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.707979 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708018 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708066 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708245 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708297 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708333 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708377 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhswp\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-kube-api-access-lhswp\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708444 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed043325-0e1b-4c86-89dc-fee8978ff145-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708496 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708547 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708861 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.708992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.708973 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.709088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.709069 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed043325-0e1b-4c86-89dc-fee8978ff145-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.710847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.710820 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.711067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.711049 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.729357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.729328 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.729525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.729507 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhswp\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-kube-api-access-lhswp\") pod \"maas-default-gateway-openshift-default-55fbd65575-slrm5\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:39.897995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:39.897925 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:40.224911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:40.224884 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5"] Oct 29 23:27:40.227313 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:27:40.227287 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded043325_0e1b_4c86_89dc_fee8978ff145.slice/crio-b1e9526f72abf69b0d875adb39a0ba156c5d909cb163284a61ec50c30453b734 WatchSource:0}: Error finding container b1e9526f72abf69b0d875adb39a0ba156c5d909cb163284a61ec50c30453b734: Status 404 returned error can't find the container with id b1e9526f72abf69b0d875adb39a0ba156c5d909cb163284a61ec50c30453b734 Oct 29 23:27:40.230290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:40.230247 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:27:40.230387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:40.230329 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:27:40.230387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:40.230362 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:27:40.517021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:40.516914 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" event={"ID":"ed043325-0e1b-4c86-89dc-fee8978ff145","Type":"ContainerStarted","Data":"4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47"} Oct 29 23:27:40.517021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:40.516983 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" event={"ID":"ed043325-0e1b-4c86-89dc-fee8978ff145","Type":"ContainerStarted","Data":"b1e9526f72abf69b0d875adb39a0ba156c5d909cb163284a61ec50c30453b734"} Oct 29 23:27:40.559186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:40.559131 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" podStartSLOduration=1.55911186 podStartE2EDuration="1.55911186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:40.556978236 +0000 UTC m=+2657.672897842" watchObservedRunningTime="2026-10-29 23:27:40.55911186 +0000 UTC m=+2657.675031468" Oct 29 23:27:40.898803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:40.898715 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:41.203319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.203229 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k"] Oct 29 23:27:41.208026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.208003 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.222790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.222764 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fb367d-openshift-default-dockercfg-6g7pj\"" Oct 29 23:27:41.230161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.230131 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzb5\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-kube-api-access-7tzb5\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.230317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.230285 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-credential-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.230393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.230351 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.230393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.230383 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf67b3ca-b596-4145-b77b-53c1207c0efd-istiod-ca-cert\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.230503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.230478 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-certs\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.230565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.230523 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-podinfo\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.230665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.230637 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-data\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.243610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.230726 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-envoy\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.243610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.230834 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-token\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.267723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.267674 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k"] Oct 29 23:27:41.355739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.355683 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-credential-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.355739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.355747 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.356017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.355774 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf67b3ca-b596-4145-b77b-53c1207c0efd-istiod-ca-cert\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.356017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.355806 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-certs\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.356017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.355835 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-podinfo\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.356017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.355880 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-data\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.356017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.355956 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-envoy\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.356282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.356076 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-token\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.356282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.356166 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-credential-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.356282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.356260 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzb5\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-kube-api-access-7tzb5\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.356518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.356483 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-socket\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.357087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.357056 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf67b3ca-b596-4145-b77b-53c1207c0efd-istiod-ca-cert\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.357694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.357635 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-data\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.357911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.357889 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-certs\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.359693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.359669 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-podinfo\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.360232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.360212 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-envoy\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.378870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.378850 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-token\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.379609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.379558 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzb5\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-kube-api-access-7tzb5\") pod \"e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.524832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.524798 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:41.922188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.922098 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-slrm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Oct 29 23:27:41.926785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:41.926738 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Oct 29 23:27:42.126863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:42.126832 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k"] Oct 29 23:27:42.129366 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:27:42.129337 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf67b3ca_b596_4145_b77b_53c1207c0efd.slice/crio-342c10e90022ba42aedc37d111b8384848f92985a3721c0d0eb38815c31f466f WatchSource:0}: Error finding container 342c10e90022ba42aedc37d111b8384848f92985a3721c0d0eb38815c31f466f: Status 404 returned error can't find the container with id 342c10e90022ba42aedc37d111b8384848f92985a3721c0d0eb38815c31f466f Oct 29 23:27:42.159354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:42.159306 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:27:42.159568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:42.159545 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:27:42.159778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:42.159756 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:27:42.647113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:42.647084 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" event={"ID":"bf67b3ca-b596-4145-b77b-53c1207c0efd","Type":"ContainerStarted","Data":"6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc"} Oct 29 23:27:42.647324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:42.647310 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" event={"ID":"bf67b3ca-b596-4145-b77b-53c1207c0efd","Type":"ContainerStarted","Data":"342c10e90022ba42aedc37d111b8384848f92985a3721c0d0eb38815c31f466f"} Oct 29 23:27:42.720806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:42.720447 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" podStartSLOduration=1.720433578 podStartE2EDuration="1.720433578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:42.717264128 +0000 UTC m=+2659.833183731" watchObservedRunningTime="2026-10-29 23:27:42.720433578 +0000 UTC m=+2659.836353183" Oct 29 23:27:42.898347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:42.898250 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-slrm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Oct 29 23:27:42.898347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:42.898328 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Oct 29 23:27:43.525195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:43.525108 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:43.526466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:43.526441 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 29 23:27:43.526598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:43.526489 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 29 23:27:43.899244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:43.899149 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-slrm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Oct 29 23:27:43.899244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:43.899211 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Oct 29 23:27:44.525830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:44.525795 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 29 23:27:44.526206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:44.525857 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 29 23:27:44.898327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:44.898248 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-slrm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Oct 29 23:27:44.898327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:44.898296 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Oct 29 23:27:45.525901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:45.525868 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 29 23:27:45.526346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:45.525921 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 29 23:27:45.899028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:45.898939 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-slrm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Oct 29 23:27:45.899028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:45.899002 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Oct 29 23:27:46.525444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:46.525413 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 29 23:27:46.525685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:46.525660 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 29 23:27:46.899328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:46.899235 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-slrm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Oct 29 23:27:46.899328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:46.899295 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Oct 29 23:27:47.525167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:47.525132 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 29 23:27:47.525431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:47.525190 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 29 23:27:47.899305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:47.899212 5736 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-slrm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Oct 29 23:27:47.899305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:47.899271 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Oct 29 23:27:48.525149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:48.525114 5736 patch_prober.go:29] interesting pod/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Oct 29 23:27:48.525348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:48.525169 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Oct 29 23:27:49.530877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:49.530842 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:49.531258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:49.531226 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:49.531949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:49.531914 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:27:49.686342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:49.686285 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:49.687440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:49.687409 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:49.688698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:27:49.688676 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:27:56.826229 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:27:56.826175 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:28:21.340284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.340249 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr"] Oct 29 23:28:21.344962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.344937 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.365245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.365217 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr"] Oct 29 23:28:21.367522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.367499 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"agc-cred-366eb7011dee\"" Oct 29 23:28:21.420314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.420276 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.420520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.420327 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.420648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.420517 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67rn\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-kube-api-access-f67rn\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.420648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.420611 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.527967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.523487 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.527967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.523547 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.527967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.524684 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67rn\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-kube-api-access-f67rn\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.527967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.524798 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.527967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.525968 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.527967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.527289 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.534102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.531562 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.552345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.551871 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67rn\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-kube-api-access-f67rn\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.658795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.658704 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:21.965623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:21.965590 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr"] Oct 29 23:28:21.980962 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:28:21.980924 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4207ef36_5fb0_4c3d_99cb_177af39d2b83.slice/crio-eb8aa564f1a815c910147aec8cccc7ee3e0ef79ddfb1c876088c37ef7af53cd7 WatchSource:0}: Error finding container eb8aa564f1a815c910147aec8cccc7ee3e0ef79ddfb1c876088c37ef7af53cd7: Status 404 returned error can't find the container with id eb8aa564f1a815c910147aec8cccc7ee3e0ef79ddfb1c876088c37ef7af53cd7 Oct 29 23:28:22.322525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:22.322437 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" event={"ID":"4207ef36-5fb0-4c3d-99cb-177af39d2b83","Type":"ContainerStarted","Data":"22d74e31f0d86f566cfca168d1b55cd0826337293a5caea9210e7e0c18916c0d"} Oct 29 23:28:22.322525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:22.322504 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" event={"ID":"4207ef36-5fb0-4c3d-99cb-177af39d2b83","Type":"ContainerStarted","Data":"eb8aa564f1a815c910147aec8cccc7ee3e0ef79ddfb1c876088c37ef7af53cd7"} Oct 29 23:28:22.322786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:22.322530 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:22.359832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:22.359786 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" podStartSLOduration=1.359773148 podStartE2EDuration="1.359773148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:22.358934905 +0000 UTC m=+2699.474854511" watchObservedRunningTime="2026-10-29 23:28:22.359773148 +0000 UTC m=+2699.475692753" Oct 29 23:28:23.893488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:23.893460 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:28:23.923363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:23.923338 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:28:26.125028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.124992 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b"] Oct 29 23:28:26.129970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.129949 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:26.147343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.147318 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0e59cd\"" Oct 29 23:28:26.162003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.161972 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b"] Oct 29 23:28:26.190745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.190702 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz"] Oct 29 23:28:26.193598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.193553 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b\" (UID: \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:26.193807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.193778 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8nz6\" (UniqueName: \"kubernetes.io/projected/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-kube-api-access-d8nz6\") pod \"payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b\" (UID: \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:26.195299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.195279 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:26.213457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.213437 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0e59cd-dockercfg-xvvg9\"" Oct 29 23:28:26.215343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.215320 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz"] Oct 29 23:28:26.321606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.320059 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b\" (UID: \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:26.321606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.320265 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8nz6\" (UniqueName: \"kubernetes.io/projected/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-kube-api-access-d8nz6\") pod \"payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b\" (UID: \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:26.321606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.320327 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf7b783-43f7-49aa-8594-8a328ece36e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz\" (UID: \"fdf7b783-43f7-49aa-8594-8a328ece36e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:26.321606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.320392 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4r9\" (UniqueName: \"kubernetes.io/projected/fdf7b783-43f7-49aa-8594-8a328ece36e0-kube-api-access-7h4r9\") pod \"payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz\" (UID: \"fdf7b783-43f7-49aa-8594-8a328ece36e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:26.445287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.424883 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf7b783-43f7-49aa-8594-8a328ece36e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz\" (UID: \"fdf7b783-43f7-49aa-8594-8a328ece36e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:26.445287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.424967 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4r9\" (UniqueName: \"kubernetes.io/projected/fdf7b783-43f7-49aa-8594-8a328ece36e0-kube-api-access-7h4r9\") pod \"payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz\" (UID: \"fdf7b783-43f7-49aa-8594-8a328ece36e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:26.445287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.427505 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf7b783-43f7-49aa-8594-8a328ece36e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz\" (UID: \"fdf7b783-43f7-49aa-8594-8a328ece36e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:26.446624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.446596 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8nz6\" (UniqueName: \"kubernetes.io/projected/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-kube-api-access-d8nz6\") pod \"payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b\" (UID: \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:26.470604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.468545 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b\" (UID: \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:26.542596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.542345 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4r9\" (UniqueName: \"kubernetes.io/projected/fdf7b783-43f7-49aa-8594-8a328ece36e0-kube-api-access-7h4r9\") pod \"payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz\" (UID: \"fdf7b783-43f7-49aa-8594-8a328ece36e0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:26.744829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.744794 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:26.811485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:26.811459 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:26.853562 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:28:26.853499 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:28:27.061378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.061343 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b"] Oct 29 23:28:27.063212 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:28:27.063182 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edd8f12_6d6a_4d3a_aec1_3a91e24c2b78.slice/crio-f438bd9cc188d49a2c21b572b0bf651d60257547dedd7a120a2129255f2dad0e WatchSource:0}: Error finding container f438bd9cc188d49a2c21b572b0bf651d60257547dedd7a120a2129255f2dad0e: Status 404 returned error can't find the container with id f438bd9cc188d49a2c21b572b0bf651d60257547dedd7a120a2129255f2dad0e Oct 29 23:28:27.106797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.106773 5736 scope.go:122] "RemoveContainer" containerID="5a1c7fecdc81bb65bf11c28a40619d3081d4d64d1f342af0c6ea512c2068fb84" Oct 29 23:28:27.117250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.117232 5736 scope.go:122] "RemoveContainer" containerID="2bd38b104d7fdf2b28fc3be39c968357a661e47d053eeb558b9ccef8be72f62a" Oct 29 23:28:27.129445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.129422 5736 scope.go:122] "RemoveContainer" containerID="1b21ba24f48e257f7866b3458d62bf5e7fcd48abe17449819279afb05ad106f8" Oct 29 23:28:27.133278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.133252 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz"] Oct 29 23:28:27.135215 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:28:27.135191 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf7b783_43f7_49aa_8594_8a328ece36e0.slice/crio-6054c17e86f07a9de954b1f45e53bc9419580018c76cf0b706c8fb4f0a0acadb WatchSource:0}: Error finding container 6054c17e86f07a9de954b1f45e53bc9419580018c76cf0b706c8fb4f0a0acadb: Status 404 returned error can't find the container with id 6054c17e86f07a9de954b1f45e53bc9419580018c76cf0b706c8fb4f0a0acadb Oct 29 23:28:27.143662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.143593 5736 scope.go:122] "RemoveContainer" containerID="5f3ac1aeff1e8546f652246be9fce6c76b8292eba02dd80311264daa3231d907" Oct 29 23:28:27.160051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.160034 5736 scope.go:122] "RemoveContainer" containerID="2421a1b92f63fce4c21ccc9ce5f50356f28646560c1b326b5069ef99a7c64ea3" Oct 29 23:28:27.170568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.170542 5736 scope.go:122] "RemoveContainer" containerID="caae1a0ffad3b25d67b8b4c4d86a9011c182e802d1b7171d835e23a228cae62a" Oct 29 23:28:27.181214 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.181194 5736 scope.go:122] "RemoveContainer" containerID="6880dc71ade6a687ed06b326a7c18343b06411766254f3b6d343df46964d9cfa" Oct 29 23:28:27.198380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.198361 5736 scope.go:122] "RemoveContainer" containerID="a144d0e99ce2a28e819abc4174837bed3ca8f93141efcce2bc17d45c2a5b7ccd" Oct 29 23:28:27.500080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.500047 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" event={"ID":"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78","Type":"ContainerStarted","Data":"12566b9ac42e864e3109359b5fe5e844cb09e3f98342263afae27f8b15cbd364"} Oct 29 23:28:27.500274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.500120 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:27.500274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.500143 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" event={"ID":"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78","Type":"ContainerStarted","Data":"f438bd9cc188d49a2c21b572b0bf651d60257547dedd7a120a2129255f2dad0e"} Oct 29 23:28:27.501859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.501829 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" event={"ID":"fdf7b783-43f7-49aa-8594-8a328ece36e0","Type":"ContainerStarted","Data":"82711292a21f775b4e056c1c854d70ce66fd8e23d53ec64a4c43cf9307044d25"} Oct 29 23:28:27.501972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.501874 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" event={"ID":"fdf7b783-43f7-49aa-8594-8a328ece36e0","Type":"ContainerStarted","Data":"6054c17e86f07a9de954b1f45e53bc9419580018c76cf0b706c8fb4f0a0acadb"} Oct 29 23:28:27.501972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.501918 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:27.539294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.539219 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" podStartSLOduration=1.539207351 podStartE2EDuration="1.539207351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:27.537756988 +0000 UTC m=+2704.653676595" watchObservedRunningTime="2026-10-29 23:28:27.539207351 +0000 UTC m=+2704.655126948" Oct 29 23:28:27.576287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:27.576240 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" podStartSLOduration=1.576226439 podStartE2EDuration="1.576226439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:27.573169036 +0000 UTC m=+2704.689088642" watchObservedRunningTime="2026-10-29 23:28:27.576226439 +0000 UTC m=+2704.692146045" Oct 29 23:28:33.336172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:33.336142 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:28:33.415510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:33.415473 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx"] Oct 29 23:28:36.032686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:36.032568 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq"] Oct 29 23:28:36.042019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:36.041993 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9905f98a-8fdnq"] Oct 29 23:28:37.005743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:37.005711 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a08b3d-3db1-40ab-a093-10f3046bcf46" path="/var/lib/kubelet/pods/32a08b3d-3db1-40ab-a093-10f3046bcf46/volumes" Oct 29 23:28:38.422733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.422650 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" podUID="7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" containerName="payload-processing" containerID="cri-o://24d945b0f6fd7cbb98351c80660e1cff0ec1dde0cc2293b36461103b6606895a" gracePeriod=25 Oct 29 23:28:38.512419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.512389 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:28:38.512839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.512548 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:28:38.616006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.615968 5736 generic.go:356] "Generic (PLEG): container finished" podID="7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" containerID="24d945b0f6fd7cbb98351c80660e1cff0ec1dde0cc2293b36461103b6606895a" exitCode=0 Oct 29 23:28:38.616401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.616373 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" event={"ID":"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e","Type":"ContainerDied","Data":"24d945b0f6fd7cbb98351c80660e1cff0ec1dde0cc2293b36461103b6606895a"} Oct 29 23:28:38.785222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.785197 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:28:38.858170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.858112 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-plugins-config-volume\") pod \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " Oct 29 23:28:38.858523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.858498 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-routing-overlay-volume\") pod \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " Oct 29 23:28:38.858693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.858539 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-plugins-config-volume" pod "7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" (UID: "7c197254-ed72-4ee2-bb1b-f09dfb29cf4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:38.858693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.858559 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-kube-api-access-lhckr\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-kube-api-access-lhckr\") pod \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " Oct 29 23:28:38.858693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.858683 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-provider-credentials\") pod \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\" (UID: \"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e\") " Oct 29 23:28:38.859271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.859246 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-routing-overlay-volume" pod "7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" (UID: "7c197254-ed72-4ee2-bb1b-f09dfb29cf4e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:38.859478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.859446 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:38.861881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.861857 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-provider-credentials" pod "7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" (UID: "7c197254-ed72-4ee2-bb1b-f09dfb29cf4e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:38.861978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.861906 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-kube-api-access-lhckr" pod "7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" (UID: "7c197254-ed72-4ee2-bb1b-f09dfb29cf4e"). InnerVolumeSpecName "kube-api-access-lhckr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:38.960637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.960597 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhckr\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-kube-api-access-lhckr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:38.960637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.960636 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:38.960846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:38.960654 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:39.625515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:39.625483 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" Oct 29 23:28:39.626024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:39.625508 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx" event={"ID":"7c197254-ed72-4ee2-bb1b-f09dfb29cf4e","Type":"ContainerDied","Data":"97d41a32efb177880be53f0dd61947c48a4f851a80294bc35ac5e4ab5467b0ea"} Oct 29 23:28:39.626024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:39.625599 5736 scope.go:122] "RemoveContainer" containerID="24d945b0f6fd7cbb98351c80660e1cff0ec1dde0cc2293b36461103b6606895a" Oct 29 23:28:39.680381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:39.680345 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx"] Oct 29 23:28:39.690612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:39.690553 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c946rxlx"] Oct 29 23:28:41.005246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:41.005214 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" path="/var/lib/kubelet/pods/7c197254-ed72-4ee2-bb1b-f09dfb29cf4e/volumes" Oct 29 23:28:44.832194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:44.832155 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk"] Oct 29 23:28:44.832624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:44.832359 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerName="istio-proxy" containerID="cri-o://16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97" gracePeriod=30 Oct 29 23:28:50.113614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.113549 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:28:50.137287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137257 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-certs\") pod \"3e7b691c-2aff-4b07-b398-90d86a6f68db\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " Oct 29 23:28:50.137406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137383 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-socket\") pod \"3e7b691c-2aff-4b07-b398-90d86a6f68db\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " Oct 29 23:28:50.137447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137427 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-kube-api-access-hbgxg\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-kube-api-access-hbgxg\") pod \"3e7b691c-2aff-4b07-b398-90d86a6f68db\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " Oct 29 23:28:50.137502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137495 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-envoy\") pod \"3e7b691c-2aff-4b07-b398-90d86a6f68db\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " Oct 29 23:28:50.137556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137520 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-certs" pod "3e7b691c-2aff-4b07-b398-90d86a6f68db" (UID: "3e7b691c-2aff-4b07-b398-90d86a6f68db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:50.137757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137563 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-token\") pod \"3e7b691c-2aff-4b07-b398-90d86a6f68db\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " Oct 29 23:28:50.137757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137653 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-socket" pod "3e7b691c-2aff-4b07-b398-90d86a6f68db" (UID: "3e7b691c-2aff-4b07-b398-90d86a6f68db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:50.137757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137677 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-data\") pod \"3e7b691c-2aff-4b07-b398-90d86a6f68db\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " Oct 29 23:28:50.137757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137728 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-credential-socket\") pod \"3e7b691c-2aff-4b07-b398-90d86a6f68db\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " Oct 29 23:28:50.138010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137768 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-podinfo\") pod \"3e7b691c-2aff-4b07-b398-90d86a6f68db\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " Oct 29 23:28:50.138010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137838 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7b691c-2aff-4b07-b398-90d86a6f68db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b691c-2aff-4b07-b398-90d86a6f68db-istiod-ca-cert\") pod \"3e7b691c-2aff-4b07-b398-90d86a6f68db\" (UID: \"3e7b691c-2aff-4b07-b398-90d86a6f68db\") " Oct 29 23:28:50.138010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137917 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-data" pod "3e7b691c-2aff-4b07-b398-90d86a6f68db" (UID: "3e7b691c-2aff-4b07-b398-90d86a6f68db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:50.138010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.137936 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-credential-socket" pod "3e7b691c-2aff-4b07-b398-90d86a6f68db" (UID: "3e7b691c-2aff-4b07-b398-90d86a6f68db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:50.138303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.138281 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7b691c-2aff-4b07-b398-90d86a6f68db-istiod-ca-cert" pod "3e7b691c-2aff-4b07-b398-90d86a6f68db" (UID: "3e7b691c-2aff-4b07-b398-90d86a6f68db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:50.138512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.138493 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:50.138622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.138512 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:50.138622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.138523 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e7b691c-2aff-4b07-b398-90d86a6f68db-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:50.138622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.138531 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:50.138622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.138541 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:50.139799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.139772 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-token" pod "3e7b691c-2aff-4b07-b398-90d86a6f68db" (UID: "3e7b691c-2aff-4b07-b398-90d86a6f68db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:50.139949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.139930 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-kube-api-access-hbgxg" pod "3e7b691c-2aff-4b07-b398-90d86a6f68db" (UID: "3e7b691c-2aff-4b07-b398-90d86a6f68db"). InnerVolumeSpecName "kube-api-access-hbgxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:50.140178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.140157 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-podinfo" pod "3e7b691c-2aff-4b07-b398-90d86a6f68db" (UID: "3e7b691c-2aff-4b07-b398-90d86a6f68db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:28:50.140224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.140160 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-envoy" pod "3e7b691c-2aff-4b07-b398-90d86a6f68db" (UID: "3e7b691c-2aff-4b07-b398-90d86a6f68db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:50.240495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.240460 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:50.240495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.240486 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:50.240495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.240495 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbgxg\" (UniqueName: \"kubernetes.io/projected/3e7b691c-2aff-4b07-b398-90d86a6f68db-kube-api-access-hbgxg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:50.240739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.240506 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e7b691c-2aff-4b07-b398-90d86a6f68db-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:28:50.726384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.726350 5736 generic.go:356] "Generic (PLEG): container finished" podID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerID="16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97" exitCode=0 Oct 29 23:28:50.726560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.726395 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" Oct 29 23:28:50.726560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.726424 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" event={"ID":"3e7b691c-2aff-4b07-b398-90d86a6f68db","Type":"ContainerDied","Data":"16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97"} Oct 29 23:28:50.726560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.726472 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk" event={"ID":"3e7b691c-2aff-4b07-b398-90d86a6f68db","Type":"ContainerDied","Data":"6b9850b1cda962f5c93eaf10ad7b51d047c36d9d76f39bd1cf8abda319853532"} Oct 29 23:28:50.726560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.726492 5736 scope.go:122] "RemoveContainer" containerID="16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97" Oct 29 23:28:50.738322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.738297 5736 scope.go:122] "RemoveContainer" containerID="16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97" Oct 29 23:28:50.738558 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:28:50.738537 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97\": container with ID starting with 16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97 not found: ID does not exist" containerID="16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97" Oct 29 23:28:50.738625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.738567 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97"} err="failed to get container status \"16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97\": rpc error: code = NotFound desc = could not find container \"16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97\": container with ID starting with 16fca3ba80a2fc4ca73ed87214c12210dfce564cb89195942ed3c8fefe9b3d97 not found: ID does not exist" Oct 29 23:28:50.786328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.786301 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk"] Oct 29 23:28:50.796632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:50.796597 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-6lqpk"] Oct 29 23:28:51.005753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:28:51.005681 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" path="/var/lib/kubelet/pods/3e7b691c-2aff-4b07-b398-90d86a6f68db/volumes" Oct 29 23:28:56.826668 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:28:56.826627 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:29:11.591128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.591086 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx"] Oct 29 23:29:11.592147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.592125 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerName="istio-proxy" Oct 29 23:29:11.592225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.592149 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerName="istio-proxy" Oct 29 23:29:11.592225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.592169 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" containerName="payload-processing" Oct 29 23:29:11.592225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.592174 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" containerName="payload-processing" Oct 29 23:29:11.592362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.592319 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7b691c-2aff-4b07-b398-90d86a6f68db" containerName="istio-proxy" Oct 29 23:29:11.592362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.592334 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c197254-ed72-4ee2-bb1b-f09dfb29cf4e" containerName="payload-processing" Oct 29 23:29:11.598017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.597991 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.615140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.615112 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"agc-cred-a2ef3b856b7b\"" Oct 29 23:29:11.624602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.621209 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"payload-processing-external-model-e2e--50b13d0a-dockercfg-lf8n6\"" Oct 29 23:29:11.624602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.621302 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a84df8\"" Oct 29 23:29:11.624602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.621447 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"routing-overlay\"" Oct 29 23:29:11.624602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.621510 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"kube-root-ca.crt\"" Oct 29 23:29:11.624602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.621562 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"openshift-service-ca.crt\"" Oct 29 23:29:11.628042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.628010 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx"] Oct 29 23:29:11.744676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.743915 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.744676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.744034 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.744676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.744120 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rpx\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-kube-api-access-85rpx\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.744676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.744279 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.847054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.846949 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.847054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.847042 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.848243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.847118 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rpx\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-kube-api-access-85rpx\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.852841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.849933 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.852841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.852072 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.856178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.853972 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.866138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.866101 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.883437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.883280 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rpx\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-kube-api-access-85rpx\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:11.922637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:11.920893 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:12.108795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.108603 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz"] Oct 29 23:29:12.114510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.114460 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.128469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.128444 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz"] Oct 29 23:29:12.133942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.133923 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"agc-cred-f2a82bc33c5a\"" Oct 29 23:29:12.160151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.160121 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.160348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.160320 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.160551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.160521 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.160635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.160573 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98sjc\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-kube-api-access-98sjc\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.242421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.242393 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx"] Oct 29 23:29:12.244723 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:12.244692 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf181d4a1_4ccc_4c15_aa6c_fd712bdfd758.slice/crio-c3d9eda0ac3e2ccdaa4f4e79a1bb9404b3b2e2f12c70461ce3a5c76d8334ee67 WatchSource:0}: Error finding container c3d9eda0ac3e2ccdaa4f4e79a1bb9404b3b2e2f12c70461ce3a5c76d8334ee67: Status 404 returned error can't find the container with id c3d9eda0ac3e2ccdaa4f4e79a1bb9404b3b2e2f12c70461ce3a5c76d8334ee67 Oct 29 23:29:12.262029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.262004 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.262184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.262135 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.262184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.262170 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sjc\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-kube-api-access-98sjc\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.262968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.262354 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.263044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.263016 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.263312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.263282 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.265073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.265055 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.283332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.283313 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sjc\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-kube-api-access-98sjc\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.434448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.434377 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:12.742876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.742850 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz"] Oct 29 23:29:12.744829 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:12.744802 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc9d562_8ded_454e_83f8_0008f4566d15.slice/crio-b0b80003c3dd8c3e14ba3bef9132966a12d6ac59ca334ccaf17726d3f69fec9c WatchSource:0}: Error finding container b0b80003c3dd8c3e14ba3bef9132966a12d6ac59ca334ccaf17726d3f69fec9c: Status 404 returned error can't find the container with id b0b80003c3dd8c3e14ba3bef9132966a12d6ac59ca334ccaf17726d3f69fec9c Oct 29 23:29:12.792922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.792897 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5"] Oct 29 23:29:12.798279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.798257 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:12.812351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.812320 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:29:12.812468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.812416 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:12.836509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.836470 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5"] Oct 29 23:29:12.872136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.872102 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a693068-df6g5\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:12.872285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.872148 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tprc2\" (UniqueName: \"kubernetes.io/projected/940a7468-b896-4faf-8f09-fcd83db78892-kube-api-access-tprc2\") pod \"maas-api-revoke-sub-8a693068-df6g5\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:12.872403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.872373 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a693068-df6g5\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:12.974099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.974070 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a693068-df6g5\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:12.974259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.974213 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a693068-df6g5\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:12.974259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.974241 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tprc2\" (UniqueName: \"kubernetes.io/projected/940a7468-b896-4faf-8f09-fcd83db78892-kube-api-access-tprc2\") pod \"maas-api-revoke-sub-8a693068-df6g5\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:12.974900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.974872 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a693068-df6g5\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:12.976362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.976339 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a693068-df6g5\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:12.995719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:12.995664 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tprc2\" (UniqueName: \"kubernetes.io/projected/940a7468-b896-4faf-8f09-fcd83db78892-kube-api-access-tprc2\") pod \"maas-api-revoke-sub-8a693068-df6g5\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:13.078439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.078382 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" event={"ID":"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758","Type":"ContainerStarted","Data":"431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c"} Oct 29 23:29:13.078685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.078470 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" event={"ID":"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758","Type":"ContainerStarted","Data":"c3d9eda0ac3e2ccdaa4f4e79a1bb9404b3b2e2f12c70461ce3a5c76d8334ee67"} Oct 29 23:29:13.081370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.079642 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:13.082308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.082281 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" event={"ID":"abc9d562-8ded-454e-83f8-0008f4566d15","Type":"ContainerStarted","Data":"88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929"} Oct 29 23:29:13.082423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.082324 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" event={"ID":"abc9d562-8ded-454e-83f8-0008f4566d15","Type":"ContainerStarted","Data":"b0b80003c3dd8c3e14ba3bef9132966a12d6ac59ca334ccaf17726d3f69fec9c"} Oct 29 23:29:13.082475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.082424 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:13.120508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.120470 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:13.121403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.121359 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" podStartSLOduration=2.121344128 podStartE2EDuration="2.121344128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:13.120560869 +0000 UTC m=+2750.236480476" watchObservedRunningTime="2026-10-29 23:29:13.121344128 +0000 UTC m=+2750.237263733" Oct 29 23:29:13.160888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.160831 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" podStartSLOduration=1.160816222 podStartE2EDuration="1.160816222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:13.158190801 +0000 UTC m=+2750.274110407" watchObservedRunningTime="2026-10-29 23:29:13.160816222 +0000 UTC m=+2750.276735828" Oct 29 23:29:13.430709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:13.430680 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5"] Oct 29 23:29:13.432396 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:13.432364 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940a7468_b896_4faf_8f09_fcd83db78892.slice/crio-5152d6664d33faeb999a59879f42f6eac93474d343913b7bf4e8f5f6b4fa45b7 WatchSource:0}: Error finding container 5152d6664d33faeb999a59879f42f6eac93474d343913b7bf4e8f5f6b4fa45b7: Status 404 returned error can't find the container with id 5152d6664d33faeb999a59879f42f6eac93474d343913b7bf4e8f5f6b4fa45b7 Oct 29 23:29:14.091497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:14.091417 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" event={"ID":"940a7468-b896-4faf-8f09-fcd83db78892","Type":"ContainerStarted","Data":"2d4ae5167948d07ed55e44d225111b2556772f361e246a3df0b27d4dcb7357b6"} Oct 29 23:29:14.091497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:14.091467 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" event={"ID":"940a7468-b896-4faf-8f09-fcd83db78892","Type":"ContainerStarted","Data":"5152d6664d33faeb999a59879f42f6eac93474d343913b7bf4e8f5f6b4fa45b7"} Oct 29 23:29:14.132292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:14.132244 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" podStartSLOduration=2.132230381 podStartE2EDuration="2.132230381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:14.128025924 +0000 UTC m=+2751.243945532" watchObservedRunningTime="2026-10-29 23:29:14.132230381 +0000 UTC m=+2751.248149987" Oct 29 23:29:14.509671 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:14.509641 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940a7468_b896_4faf_8f09_fcd83db78892.slice/crio-conmon-2d4ae5167948d07ed55e44d225111b2556772f361e246a3df0b27d4dcb7357b6.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:29:15.099765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:15.099726 5736 generic.go:356] "Generic (PLEG): container finished" podID="940a7468-b896-4faf-8f09-fcd83db78892" containerID="2d4ae5167948d07ed55e44d225111b2556772f361e246a3df0b27d4dcb7357b6" exitCode=0 Oct 29 23:29:15.100167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:15.099807 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" event={"ID":"940a7468-b896-4faf-8f09-fcd83db78892","Type":"ContainerDied","Data":"2d4ae5167948d07ed55e44d225111b2556772f361e246a3df0b27d4dcb7357b6"} Oct 29 23:29:16.277599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.277548 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:16.361001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.360917 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-xks-ca\") pod \"940a7468-b896-4faf-8f09-fcd83db78892\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " Oct 29 23:29:16.361180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.361004 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940a7468-b896-4faf-8f09-fcd83db78892-kube-api-access-tprc2\" (UniqueName: \"kubernetes.io/projected/940a7468-b896-4faf-8f09-fcd83db78892-kube-api-access-tprc2\") pod \"940a7468-b896-4faf-8f09-fcd83db78892\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " Oct 29 23:29:16.361180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.361030 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-openshift-ca\") pod \"940a7468-b896-4faf-8f09-fcd83db78892\" (UID: \"940a7468-b896-4faf-8f09-fcd83db78892\") " Oct 29 23:29:16.361742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.361710 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-openshift-ca" pod "940a7468-b896-4faf-8f09-fcd83db78892" (UID: "940a7468-b896-4faf-8f09-fcd83db78892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:16.363800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.363773 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-xks-ca" pod "940a7468-b896-4faf-8f09-fcd83db78892" (UID: "940a7468-b896-4faf-8f09-fcd83db78892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:16.363930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.363831 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940a7468-b896-4faf-8f09-fcd83db78892-kube-api-access-tprc2" pod "940a7468-b896-4faf-8f09-fcd83db78892" (UID: "940a7468-b896-4faf-8f09-fcd83db78892"). InnerVolumeSpecName "kube-api-access-tprc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:16.463429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.463399 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:16.463429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.463427 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tprc2\" (UniqueName: \"kubernetes.io/projected/940a7468-b896-4faf-8f09-fcd83db78892-kube-api-access-tprc2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:16.463619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.463437 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940a7468-b896-4faf-8f09-fcd83db78892-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:16.945179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:16.944144 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx"] Oct 29 23:29:17.121845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:17.121809 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" Oct 29 23:29:17.122056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:17.122025 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5" event={"ID":"940a7468-b896-4faf-8f09-fcd83db78892","Type":"ContainerDied","Data":"5152d6664d33faeb999a59879f42f6eac93474d343913b7bf4e8f5f6b4fa45b7"} Oct 29 23:29:17.122170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:17.122081 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5152d6664d33faeb999a59879f42f6eac93474d343913b7bf4e8f5f6b4fa45b7" Oct 29 23:29:19.924374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.924345 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6"] Oct 29 23:29:19.925200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.925182 5736 cpu_manager.go:414] "containerMap: removing container" podUID="940a7468-b896-4faf-8f09-fcd83db78892" containerName="revoke-keys" Oct 29 23:29:19.925249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.925200 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940a7468-b896-4faf-8f09-fcd83db78892" containerName="revoke-keys" Oct 29 23:29:19.925373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.925362 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="940a7468-b896-4faf-8f09-fcd83db78892" containerName="revoke-keys" Oct 29 23:29:19.929329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.929308 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:19.953936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.953907 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6f162a\"" Oct 29 23:29:19.954088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.953947 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"agc-cred-1298ae2d22f5\"" Oct 29 23:29:19.954088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.953951 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"openshift-service-ca.crt\"" Oct 29 23:29:19.954088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.953997 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"routing-overlay\"" Oct 29 23:29:19.954221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.954091 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"kube-root-ca.crt\"" Oct 29 23:29:19.954281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.954216 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"payload-processing-external-model-e2e--b1ab4497-dockercfg-h4qm8\"" Oct 29 23:29:19.960480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.960074 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6"] Oct 29 23:29:19.977193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.976244 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:19.977193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.976347 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:19.977193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.976466 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:19.977193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:19.976738 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7h6m\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-kube-api-access-p7h6m\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.113601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.104345 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7h6m\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-kube-api-access-p7h6m\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.113601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.104470 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.113601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.104615 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.113601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.104786 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.113601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.111612 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.232631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.231531 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.233729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.233686 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.242932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.242874 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7h6m\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-kube-api-access-p7h6m\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.244791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.244512 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:20.355621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.354435 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl"] Oct 29 23:29:20.366845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.364322 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.378684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.378653 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"agc-cred-f58eed665d85\"" Oct 29 23:29:20.391636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.391463 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl"] Oct 29 23:29:20.423160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.423122 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.423339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.423313 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6kwb\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-kube-api-access-k6kwb\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.423410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.423380 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.423471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.423445 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.525453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.525374 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.525629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.525497 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6kwb\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-kube-api-access-k6kwb\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.525629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.525536 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.525629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.525594 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.526364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.526326 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.527115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.527091 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.528071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.528051 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.547098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.547069 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6kwb\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-kube-api-access-k6kwb\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:20.573132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.573110 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6"] Oct 29 23:29:20.575274 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:20.575251 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode811fbc8_69f4_4996_b14e_04aa58317174.slice/crio-b234ab459e01e7e0af61306b807e5d7bf6527afd1390c34144aec9dcff1dfa42 WatchSource:0}: Error finding container b234ab459e01e7e0af61306b807e5d7bf6527afd1390c34144aec9dcff1dfa42: Status 404 returned error can't find the container with id b234ab459e01e7e0af61306b807e5d7bf6527afd1390c34144aec9dcff1dfa42 Oct 29 23:29:20.699543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:20.699514 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:21.009501 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:21.009468 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34733fb2_3013_4eb8_9d44_5b3be1b3a263.slice/crio-a4c712b809d472f48aeafdfb14a4bff7322d3cf24c823ea5e1bf93963e97b7c9 WatchSource:0}: Error finding container a4c712b809d472f48aeafdfb14a4bff7322d3cf24c823ea5e1bf93963e97b7c9: Status 404 returned error can't find the container with id a4c712b809d472f48aeafdfb14a4bff7322d3cf24c823ea5e1bf93963e97b7c9 Oct 29 23:29:21.009856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.009653 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl"] Oct 29 23:29:21.261123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.261016 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" event={"ID":"e811fbc8-69f4-4996-b14e-04aa58317174","Type":"ContainerStarted","Data":"9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604"} Oct 29 23:29:21.261123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.261087 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" event={"ID":"e811fbc8-69f4-4996-b14e-04aa58317174","Type":"ContainerStarted","Data":"b234ab459e01e7e0af61306b807e5d7bf6527afd1390c34144aec9dcff1dfa42"} Oct 29 23:29:21.261378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.261354 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:21.263286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.263259 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" event={"ID":"34733fb2-3013-4eb8-9d44-5b3be1b3a263","Type":"ContainerStarted","Data":"825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d"} Oct 29 23:29:21.263408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.263301 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" event={"ID":"34733fb2-3013-4eb8-9d44-5b3be1b3a263","Type":"ContainerStarted","Data":"a4c712b809d472f48aeafdfb14a4bff7322d3cf24c823ea5e1bf93963e97b7c9"} Oct 29 23:29:21.263408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.263333 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:21.303260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.303217 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" podStartSLOduration=2.3032049150000002 podStartE2EDuration="2.303204915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:21.299051558 +0000 UTC m=+2758.414971163" watchObservedRunningTime="2026-10-29 23:29:21.303204915 +0000 UTC m=+2758.419124521" Oct 29 23:29:21.337279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.337226 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" podStartSLOduration=1.33720948 podStartE2EDuration="1.33720948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:21.335927177 +0000 UTC m=+2758.451846783" watchObservedRunningTime="2026-10-29 23:29:21.33720948 +0000 UTC m=+2758.453129087" Oct 29 23:29:21.962608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:21.957645 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" podUID="f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" containerName="payload-processing" containerID="cri-o://431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c" gracePeriod=25 Oct 29 23:29:22.252349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.252321 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:22.271821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.271795 5736 generic.go:356] "Generic (PLEG): container finished" podID="f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" containerID="431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c" exitCode=0 Oct 29 23:29:22.271953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.271838 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" Oct 29 23:29:22.271953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.271876 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" event={"ID":"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758","Type":"ContainerDied","Data":"431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c"} Oct 29 23:29:22.271953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.271929 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx" event={"ID":"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758","Type":"ContainerDied","Data":"c3d9eda0ac3e2ccdaa4f4e79a1bb9404b3b2e2f12c70461ce3a5c76d8334ee67"} Oct 29 23:29:22.271953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.271946 5736 scope.go:122] "RemoveContainer" containerID="431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c" Oct 29 23:29:22.282713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.282692 5736 scope.go:122] "RemoveContainer" containerID="431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c" Oct 29 23:29:22.282961 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:22.282941 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c\": container with ID starting with 431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c not found: ID does not exist" containerID="431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c" Oct 29 23:29:22.283012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.282967 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c"} err="failed to get container status \"431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c\": rpc error: code = NotFound desc = could not find container \"431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c\": container with ID starting with 431d67d120c916747c2e3945978a1b08bc017f6789e1f8163086b184f592cd1c not found: ID does not exist" Oct 29 23:29:22.364573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.364545 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-plugins-config-volume\") pod \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " Oct 29 23:29:22.364768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.364698 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-provider-credentials\") pod \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " Oct 29 23:29:22.364768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.364759 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-kube-api-access-85rpx\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-kube-api-access-85rpx\") pod \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " Oct 29 23:29:22.364892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.364842 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-routing-overlay-volume\") pod \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\" (UID: \"f181d4a1-4ccc-4c15-aa6c-fd712bdfd758\") " Oct 29 23:29:22.364961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.364941 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-plugins-config-volume" pod "f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" (UID: "f181d4a1-4ccc-4c15-aa6c-fd712bdfd758"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:22.365321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.365273 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-routing-overlay-volume" pod "f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" (UID: "f181d4a1-4ccc-4c15-aa6c-fd712bdfd758"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:22.365837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.365810 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:22.365949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.365839 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:22.367285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.367261 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-provider-credentials" pod "f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" (UID: "f181d4a1-4ccc-4c15-aa6c-fd712bdfd758"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:22.367406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.367386 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-kube-api-access-85rpx" pod "f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" (UID: "f181d4a1-4ccc-4c15-aa6c-fd712bdfd758"). InnerVolumeSpecName "kube-api-access-85rpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:22.466936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.466910 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:22.466936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.466934 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rpx\" (UniqueName: \"kubernetes.io/projected/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758-kube-api-access-85rpx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:22.616728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.616699 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx"] Oct 29 23:29:22.626570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:22.626539 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-757cn4dgx"] Oct 29 23:29:23.007053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.007018 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" path="/var/lib/kubelet/pods/f181d4a1-4ccc-4c15-aa6c-fd712bdfd758/volumes" Oct 29 23:29:23.613485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.613449 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9"] Oct 29 23:29:23.614240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.614223 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" containerName="payload-processing" Oct 29 23:29:23.614289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.614241 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" containerName="payload-processing" Oct 29 23:29:23.614366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.614356 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f181d4a1-4ccc-4c15-aa6c-fd712bdfd758" containerName="payload-processing" Oct 29 23:29:23.618986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.618962 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.638281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.638250 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9"] Oct 29 23:29:23.684748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.684713 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-880e1e95-5mlx9\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.684909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.684878 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-880e1e95-5mlx9\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.685013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.684992 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf26g\" (UniqueName: \"kubernetes.io/projected/3b82e211-44f1-4599-b15e-07aa062fb35f-kube-api-access-tf26g\") pod \"maas-api-revoke-sub-880e1e95-5mlx9\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.705980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.705955 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:23.706125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.706106 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:29:23.786880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.786845 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-880e1e95-5mlx9\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.787038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.786986 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-880e1e95-5mlx9\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.787099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.787061 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf26g\" (UniqueName: \"kubernetes.io/projected/3b82e211-44f1-4599-b15e-07aa062fb35f-kube-api-access-tf26g\") pod \"maas-api-revoke-sub-880e1e95-5mlx9\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.787779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.787748 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-880e1e95-5mlx9\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.789893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.789861 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-880e1e95-5mlx9\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.808933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.808908 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf26g\" (UniqueName: \"kubernetes.io/projected/3b82e211-44f1-4599-b15e-07aa062fb35f-kube-api-access-tf26g\") pod \"maas-api-revoke-sub-880e1e95-5mlx9\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:23.935332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:23.935272 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:24.094319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:24.094291 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:29:24.169296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:24.169258 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr"] Oct 29 23:29:24.247387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:24.247361 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9"] Oct 29 23:29:24.250072 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:24.250042 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b82e211_44f1_4599_b15e_07aa062fb35f.slice/crio-57664e66cdfb82e9138edc982710c1372b9146c5cc43fe9bd58dd5d905d513b2 WatchSource:0}: Error finding container 57664e66cdfb82e9138edc982710c1372b9146c5cc43fe9bd58dd5d905d513b2: Status 404 returned error can't find the container with id 57664e66cdfb82e9138edc982710c1372b9146c5cc43fe9bd58dd5d905d513b2 Oct 29 23:29:24.294645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:24.294610 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" event={"ID":"3b82e211-44f1-4599-b15e-07aa062fb35f","Type":"ContainerStarted","Data":"57664e66cdfb82e9138edc982710c1372b9146c5cc43fe9bd58dd5d905d513b2"} Oct 29 23:29:25.304473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:25.304440 5736 generic.go:356] "Generic (PLEG): container finished" podID="3b82e211-44f1-4599-b15e-07aa062fb35f" containerID="7d6242c5355425ffbf6de37897fadf126039132c5a5a8862239b0d6cab402677" exitCode=0 Oct 29 23:29:25.304921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:25.304530 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" event={"ID":"3b82e211-44f1-4599-b15e-07aa062fb35f","Type":"ContainerDied","Data":"7d6242c5355425ffbf6de37897fadf126039132c5a5a8862239b0d6cab402677"} Oct 29 23:29:25.563383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:25.562163 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl"] Oct 29 23:29:26.038603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.038202 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6"] Oct 29 23:29:26.059656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.045454 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:26.773822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.773788 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:26.827129 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:26.827088 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:29:26.827555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.827532 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-openshift-ca\") pod \"3b82e211-44f1-4599-b15e-07aa062fb35f\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " Oct 29 23:29:26.827694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.827638 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b82e211-44f1-4599-b15e-07aa062fb35f-kube-api-access-tf26g\" (UniqueName: \"kubernetes.io/projected/3b82e211-44f1-4599-b15e-07aa062fb35f-kube-api-access-tf26g\") pod \"3b82e211-44f1-4599-b15e-07aa062fb35f\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " Oct 29 23:29:26.827694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.827677 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-xks-ca\") pod \"3b82e211-44f1-4599-b15e-07aa062fb35f\" (UID: \"3b82e211-44f1-4599-b15e-07aa062fb35f\") " Oct 29 23:29:26.827973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.827949 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-openshift-ca" pod "3b82e211-44f1-4599-b15e-07aa062fb35f" (UID: "3b82e211-44f1-4599-b15e-07aa062fb35f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:26.828467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.828447 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:26.829844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.829815 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-xks-ca" pod "3b82e211-44f1-4599-b15e-07aa062fb35f" (UID: "3b82e211-44f1-4599-b15e-07aa062fb35f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:26.830246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.830225 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b82e211-44f1-4599-b15e-07aa062fb35f-kube-api-access-tf26g" pod "3b82e211-44f1-4599-b15e-07aa062fb35f" (UID: "3b82e211-44f1-4599-b15e-07aa062fb35f"). InnerVolumeSpecName "kube-api-access-tf26g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:26.930345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.930283 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf26g\" (UniqueName: \"kubernetes.io/projected/3b82e211-44f1-4599-b15e-07aa062fb35f-kube-api-access-tf26g\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:26.930345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:26.930313 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b82e211-44f1-4599-b15e-07aa062fb35f-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:27.322731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:27.322707 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" Oct 29 23:29:27.322836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:27.322702 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9" event={"ID":"3b82e211-44f1-4599-b15e-07aa062fb35f","Type":"ContainerDied","Data":"57664e66cdfb82e9138edc982710c1372b9146c5cc43fe9bd58dd5d905d513b2"} Oct 29 23:29:27.322836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:27.322831 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57664e66cdfb82e9138edc982710c1372b9146c5cc43fe9bd58dd5d905d513b2" Oct 29 23:29:28.182211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.182183 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82"] Oct 29 23:29:28.183064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.183045 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3b82e211-44f1-4599-b15e-07aa062fb35f" containerName="revoke-keys" Oct 29 23:29:28.183120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.183066 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b82e211-44f1-4599-b15e-07aa062fb35f" containerName="revoke-keys" Oct 29 23:29:28.183187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.183176 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b82e211-44f1-4599-b15e-07aa062fb35f" containerName="revoke-keys" Oct 29 23:29:28.188512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.188491 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.207633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.207607 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82"] Oct 29 23:29:28.212409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.212383 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0e59cd-serving-cert\"" Oct 29 23:29:28.255853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.255820 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.256138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.256116 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f74d3-9981-4244-a824-ee910022a624-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.256247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.256224 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk96v\" (UniqueName: \"kubernetes.io/projected/6e1f74d3-9981-4244-a824-ee910022a624-kube-api-access-zk96v\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.256324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.256293 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.358283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.358249 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.358512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.358484 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f74d3-9981-4244-a824-ee910022a624-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.358596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.358559 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk96v\" (UniqueName: \"kubernetes.io/projected/6e1f74d3-9981-4244-a824-ee910022a624-kube-api-access-zk96v\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.358656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.358628 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.359354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.359327 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f74d3-9981-4244-a824-ee910022a624-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.360786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.360764 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.361118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.361092 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.380397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.380370 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk96v\" (UniqueName: \"kubernetes.io/projected/6e1f74d3-9981-4244-a824-ee910022a624-kube-api-access-zk96v\") pod \"maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.503742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.503712 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:28.917199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.917173 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82"] Oct 29 23:29:28.919154 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:28.919124 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1f74d3_9981_4244_a824_ee910022a624.slice/crio-ce1630fb88305498b30d2db3abf2e853ebdc61d415429b707a945c52ce9be2f2 WatchSource:0}: Error finding container ce1630fb88305498b30d2db3abf2e853ebdc61d415429b707a945c52ce9be2f2: Status 404 returned error can't find the container with id ce1630fb88305498b30d2db3abf2e853ebdc61d415429b707a945c52ce9be2f2 Oct 29 23:29:28.920715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:28.920695 5736 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:29:29.176755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.176651 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" podUID="4207ef36-5fb0-4c3d-99cb-177af39d2b83" containerName="payload-processing" containerID="cri-o://22d74e31f0d86f566cfca168d1b55cd0826337293a5caea9210e7e0c18916c0d" gracePeriod=25 Oct 29 23:29:29.341294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.341209 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" event={"ID":"6e1f74d3-9981-4244-a824-ee910022a624","Type":"ContainerStarted","Data":"ce1630fb88305498b30d2db3abf2e853ebdc61d415429b707a945c52ce9be2f2"} Oct 29 23:29:29.343415 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.343063 5736 generic.go:356] "Generic (PLEG): container finished" podID="4207ef36-5fb0-4c3d-99cb-177af39d2b83" containerID="22d74e31f0d86f566cfca168d1b55cd0826337293a5caea9210e7e0c18916c0d" exitCode=0 Oct 29 23:29:29.343415 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.343341 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" event={"ID":"4207ef36-5fb0-4c3d-99cb-177af39d2b83","Type":"ContainerDied","Data":"22d74e31f0d86f566cfca168d1b55cd0826337293a5caea9210e7e0c18916c0d"} Oct 29 23:29:29.710637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.710611 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:29:29.791725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.791649 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-provider-credentials\") pod \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " Oct 29 23:29:29.791914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.791870 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-kube-api-access-f67rn\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-kube-api-access-f67rn\") pod \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " Oct 29 23:29:29.794621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.791993 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-routing-overlay-volume\") pod \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " Oct 29 23:29:29.794621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.792191 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-plugins-config-volume\") pod \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\" (UID: \"4207ef36-5fb0-4c3d-99cb-177af39d2b83\") " Oct 29 23:29:29.794621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.793257 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-plugins-config-volume" pod "4207ef36-5fb0-4c3d-99cb-177af39d2b83" (UID: "4207ef36-5fb0-4c3d-99cb-177af39d2b83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:29.794621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.793665 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-routing-overlay-volume" pod "4207ef36-5fb0-4c3d-99cb-177af39d2b83" (UID: "4207ef36-5fb0-4c3d-99cb-177af39d2b83"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:29.795342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.795311 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-provider-credentials" pod "4207ef36-5fb0-4c3d-99cb-177af39d2b83" (UID: "4207ef36-5fb0-4c3d-99cb-177af39d2b83"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:29.799434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.799409 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-kube-api-access-f67rn" pod "4207ef36-5fb0-4c3d-99cb-177af39d2b83" (UID: "4207ef36-5fb0-4c3d-99cb-177af39d2b83"). InnerVolumeSpecName "kube-api-access-f67rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:29.894417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.894371 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67rn\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-kube-api-access-f67rn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:29.894417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.894413 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:29.894417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.894423 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207ef36-5fb0-4c3d-99cb-177af39d2b83-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:29.894417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.894432 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4207ef36-5fb0-4c3d-99cb-177af39d2b83-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:29.994920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.994886 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5"] Oct 29 23:29:29.995646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.995625 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4207ef36-5fb0-4c3d-99cb-177af39d2b83" containerName="payload-processing" Oct 29 23:29:29.995646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.995642 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4207ef36-5fb0-4c3d-99cb-177af39d2b83" containerName="payload-processing" Oct 29 23:29:29.995941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:29.995925 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4207ef36-5fb0-4c3d-99cb-177af39d2b83" containerName="payload-processing" Oct 29 23:29:30.000114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.000088 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.023919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.022935 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:30.029386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.029350 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5"] Oct 29 23:29:30.032949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.032922 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:29:30.096922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.096891 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70cbd544-bwqq5\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.097130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.097106 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70cbd544-bwqq5\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.097235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.097219 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdnb\" (UniqueName: \"kubernetes.io/projected/9a87911d-6962-44f8-9307-5b8a70dab36c-kube-api-access-mzdnb\") pod \"maas-api-revoke-sub-70cbd544-bwqq5\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.199091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.199065 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70cbd544-bwqq5\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.199242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.199157 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdnb\" (UniqueName: \"kubernetes.io/projected/9a87911d-6962-44f8-9307-5b8a70dab36c-kube-api-access-mzdnb\") pod \"maas-api-revoke-sub-70cbd544-bwqq5\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.199310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.199298 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70cbd544-bwqq5\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.199807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.199782 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70cbd544-bwqq5\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.201673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.201655 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70cbd544-bwqq5\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.220890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.220868 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdnb\" (UniqueName: \"kubernetes.io/projected/9a87911d-6962-44f8-9307-5b8a70dab36c-kube-api-access-mzdnb\") pod \"maas-api-revoke-sub-70cbd544-bwqq5\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.339253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.339187 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:30.358642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.358620 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" Oct 29 23:29:30.359019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.358618 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr" event={"ID":"4207ef36-5fb0-4c3d-99cb-177af39d2b83","Type":"ContainerDied","Data":"eb8aa564f1a815c910147aec8cccc7ee3e0ef79ddfb1c876088c37ef7af53cd7"} Oct 29 23:29:30.359019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.358752 5736 scope.go:122] "RemoveContainer" containerID="22d74e31f0d86f566cfca168d1b55cd0826337293a5caea9210e7e0c18916c0d" Oct 29 23:29:30.361119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.361091 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" event={"ID":"6e1f74d3-9981-4244-a824-ee910022a624","Type":"ContainerStarted","Data":"6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48"} Oct 29 23:29:30.361684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.361667 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:30.463429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.462971 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" podStartSLOduration=2.091506632 podStartE2EDuration="2.462952866s" totalImagesPullingTime="371.446234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:30.422739919 +0000 UTC m=+2767.538659526" watchObservedRunningTime="2026-10-29 23:29:30.462952866 +0000 UTC m=+2767.578872475" Oct 29 23:29:30.467102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.465825 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr"] Oct 29 23:29:30.472301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.471862 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc7lhsgr"] Oct 29 23:29:30.567668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.566753 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" podUID="34733fb2-3013-4eb8-9d44-5b3be1b3a263" containerName="payload-processing" containerID="cri-o://825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d" gracePeriod=25 Oct 29 23:29:30.666792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.666759 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5"] Oct 29 23:29:30.856096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.856074 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:30.914999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.914966 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-routing-overlay-volume\") pod \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " Oct 29 23:29:30.915170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.915063 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-provider-credentials\") pod \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " Oct 29 23:29:30.915245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.915179 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-kube-api-access-k6kwb\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-kube-api-access-k6kwb\") pod \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " Oct 29 23:29:30.915343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.915315 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-routing-overlay-volume" pod "34733fb2-3013-4eb8-9d44-5b3be1b3a263" (UID: "34733fb2-3013-4eb8-9d44-5b3be1b3a263"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:30.915439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.915419 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-plugins-config-volume\") pod \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\" (UID: \"34733fb2-3013-4eb8-9d44-5b3be1b3a263\") " Oct 29 23:29:30.915812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.915792 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-plugins-config-volume" pod "34733fb2-3013-4eb8-9d44-5b3be1b3a263" (UID: "34733fb2-3013-4eb8-9d44-5b3be1b3a263"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:30.916149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.916132 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:30.916241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.916150 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34733fb2-3013-4eb8-9d44-5b3be1b3a263-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:30.918081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.918060 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-kube-api-access-k6kwb" pod "34733fb2-3013-4eb8-9d44-5b3be1b3a263" (UID: "34733fb2-3013-4eb8-9d44-5b3be1b3a263"). InnerVolumeSpecName "kube-api-access-k6kwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:30.918185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:30.918090 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-provider-credentials" pod "34733fb2-3013-4eb8-9d44-5b3be1b3a263" (UID: "34733fb2-3013-4eb8-9d44-5b3be1b3a263"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:31.015501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.015467 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4207ef36-5fb0-4c3d-99cb-177af39d2b83" path="/var/lib/kubelet/pods/4207ef36-5fb0-4c3d-99cb-177af39d2b83/volumes" Oct 29 23:29:31.017368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.017338 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6kwb\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-kube-api-access-k6kwb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:31.017482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.017371 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34733fb2-3013-4eb8-9d44-5b3be1b3a263-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:31.053740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.053709 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" podUID="e811fbc8-69f4-4996-b14e-04aa58317174" containerName="payload-processing" containerID="cri-o://9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604" gracePeriod=25 Oct 29 23:29:31.307771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.307741 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:31.321106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.321079 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-plugins-config-volume\") pod \"e811fbc8-69f4-4996-b14e-04aa58317174\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " Oct 29 23:29:31.321247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.321156 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-provider-credentials\") pod \"e811fbc8-69f4-4996-b14e-04aa58317174\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " Oct 29 23:29:31.321322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.321262 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-kube-api-access-p7h6m\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-kube-api-access-p7h6m\") pod \"e811fbc8-69f4-4996-b14e-04aa58317174\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " Oct 29 23:29:31.321520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.321492 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-plugins-config-volume" pod "e811fbc8-69f4-4996-b14e-04aa58317174" (UID: "e811fbc8-69f4-4996-b14e-04aa58317174"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:31.321987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.321963 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:31.323276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.323251 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-provider-credentials" pod "e811fbc8-69f4-4996-b14e-04aa58317174" (UID: "e811fbc8-69f4-4996-b14e-04aa58317174"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:31.323352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.323339 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-kube-api-access-p7h6m" pod "e811fbc8-69f4-4996-b14e-04aa58317174" (UID: "e811fbc8-69f4-4996-b14e-04aa58317174"). InnerVolumeSpecName "kube-api-access-p7h6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:31.370860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.370826 5736 generic.go:356] "Generic (PLEG): container finished" podID="e811fbc8-69f4-4996-b14e-04aa58317174" containerID="9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604" exitCode=0 Oct 29 23:29:31.371292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.370864 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" Oct 29 23:29:31.371292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.370985 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" event={"ID":"e811fbc8-69f4-4996-b14e-04aa58317174","Type":"ContainerDied","Data":"9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604"} Oct 29 23:29:31.371292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.371032 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6" event={"ID":"e811fbc8-69f4-4996-b14e-04aa58317174","Type":"ContainerDied","Data":"b234ab459e01e7e0af61306b807e5d7bf6527afd1390c34144aec9dcff1dfa42"} Oct 29 23:29:31.371292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.371063 5736 scope.go:122] "RemoveContainer" containerID="9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604" Oct 29 23:29:31.374882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.374855 5736 generic.go:356] "Generic (PLEG): container finished" podID="34733fb2-3013-4eb8-9d44-5b3be1b3a263" containerID="825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d" exitCode=0 Oct 29 23:29:31.375004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.374898 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" Oct 29 23:29:31.375066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.374998 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" event={"ID":"34733fb2-3013-4eb8-9d44-5b3be1b3a263","Type":"ContainerDied","Data":"825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d"} Oct 29 23:29:31.375066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.375040 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl" event={"ID":"34733fb2-3013-4eb8-9d44-5b3be1b3a263","Type":"ContainerDied","Data":"a4c712b809d472f48aeafdfb14a4bff7322d3cf24c823ea5e1bf93963e97b7c9"} Oct 29 23:29:31.377673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.377626 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" event={"ID":"9a87911d-6962-44f8-9307-5b8a70dab36c","Type":"ContainerStarted","Data":"54a1ab8c688a983b2839fc51205e40b757a305fde139a55bc867de27738c9ddc"} Oct 29 23:29:31.377673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.377676 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" event={"ID":"9a87911d-6962-44f8-9307-5b8a70dab36c","Type":"ContainerStarted","Data":"17abcc4ba17e1cfd75d0bee0ba60e599701c980e93706c0b03876e9039a95ba5"} Oct 29 23:29:31.414241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.414001 5736 scope.go:122] "RemoveContainer" containerID="9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604" Oct 29 23:29:31.414786 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:31.414488 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604\": container with ID starting with 9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604 not found: ID does not exist" containerID="9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604" Oct 29 23:29:31.414786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.414527 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604"} err="failed to get container status \"9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604\": rpc error: code = NotFound desc = could not find container \"9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604\": container with ID starting with 9bddc5b7e9b3fdebae66feae0f3c5c06eb607492d545ad821784024edcb2b604 not found: ID does not exist" Oct 29 23:29:31.414786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.414547 5736 scope.go:122] "RemoveContainer" containerID="825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d" Oct 29 23:29:31.423288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.423258 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-routing-overlay-volume\") pod \"e811fbc8-69f4-4996-b14e-04aa58317174\" (UID: \"e811fbc8-69f4-4996-b14e-04aa58317174\") " Oct 29 23:29:31.423832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.423810 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:31.423908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.423839 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7h6m\" (UniqueName: \"kubernetes.io/projected/e811fbc8-69f4-4996-b14e-04aa58317174-kube-api-access-p7h6m\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:31.424254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.424236 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-routing-overlay-volume" pod "e811fbc8-69f4-4996-b14e-04aa58317174" (UID: "e811fbc8-69f4-4996-b14e-04aa58317174"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:31.427809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.427790 5736 scope.go:122] "RemoveContainer" containerID="825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d" Oct 29 23:29:31.428069 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:31.428049 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d\": container with ID starting with 825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d not found: ID does not exist" containerID="825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d" Oct 29 23:29:31.428130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.428077 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d"} err="failed to get container status \"825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d\": rpc error: code = NotFound desc = could not find container \"825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d\": container with ID starting with 825fb5240c15b7de32a099be75eb760b91d8d30d37b8312aa410e056e38b3d2d not found: ID does not exist" Oct 29 23:29:31.441194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.441145 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" podStartSLOduration=2.441133443 podStartE2EDuration="2.441133443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:31.438103273 +0000 UTC m=+2768.554022880" watchObservedRunningTime="2026-10-29 23:29:31.441133443 +0000 UTC m=+2768.557053049" Oct 29 23:29:31.473453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.473430 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl"] Oct 29 23:29:31.476905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.476883 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-7cd79n8jl"] Oct 29 23:29:31.526089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.526063 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e811fbc8-69f4-4996-b14e-04aa58317174-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:31.715284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.715257 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6"] Oct 29 23:29:31.721302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:31.721276 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-5ccdcv7v6"] Oct 29 23:29:32.417775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:32.417734 5736 generic.go:356] "Generic (PLEG): container finished" podID="9a87911d-6962-44f8-9307-5b8a70dab36c" containerID="54a1ab8c688a983b2839fc51205e40b757a305fde139a55bc867de27738c9ddc" exitCode=0 Oct 29 23:29:32.418171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:32.417855 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" event={"ID":"9a87911d-6962-44f8-9307-5b8a70dab36c","Type":"ContainerDied","Data":"54a1ab8c688a983b2839fc51205e40b757a305fde139a55bc867de27738c9ddc"} Oct 29 23:29:33.007911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.007871 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34733fb2-3013-4eb8-9d44-5b3be1b3a263" path="/var/lib/kubelet/pods/34733fb2-3013-4eb8-9d44-5b3be1b3a263/volumes" Oct 29 23:29:33.008388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.008369 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e811fbc8-69f4-4996-b14e-04aa58317174" path="/var/lib/kubelet/pods/e811fbc8-69f4-4996-b14e-04aa58317174/volumes" Oct 29 23:29:33.601916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.601885 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:33.661517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.661484 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a87911d-6962-44f8-9307-5b8a70dab36c-kube-api-access-mzdnb\" (UniqueName: \"kubernetes.io/projected/9a87911d-6962-44f8-9307-5b8a70dab36c-kube-api-access-mzdnb\") pod \"9a87911d-6962-44f8-9307-5b8a70dab36c\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " Oct 29 23:29:33.661726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.661654 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-openshift-ca\") pod \"9a87911d-6962-44f8-9307-5b8a70dab36c\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " Oct 29 23:29:33.662004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.661980 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-xks-ca\") pod \"9a87911d-6962-44f8-9307-5b8a70dab36c\" (UID: \"9a87911d-6962-44f8-9307-5b8a70dab36c\") " Oct 29 23:29:33.662337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.662307 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-openshift-ca" pod "9a87911d-6962-44f8-9307-5b8a70dab36c" (UID: "9a87911d-6962-44f8-9307-5b8a70dab36c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:33.662747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.662707 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:33.664318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.664292 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a87911d-6962-44f8-9307-5b8a70dab36c-kube-api-access-mzdnb" pod "9a87911d-6962-44f8-9307-5b8a70dab36c" (UID: "9a87911d-6962-44f8-9307-5b8a70dab36c"). InnerVolumeSpecName "kube-api-access-mzdnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:33.664834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.664811 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-xks-ca" pod "9a87911d-6962-44f8-9307-5b8a70dab36c" (UID: "9a87911d-6962-44f8-9307-5b8a70dab36c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:33.764558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.764526 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzdnb\" (UniqueName: \"kubernetes.io/projected/9a87911d-6962-44f8-9307-5b8a70dab36c-kube-api-access-mzdnb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:33.764558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:33.764553 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a87911d-6962-44f8-9307-5b8a70dab36c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:34.438801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:34.438762 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" Oct 29 23:29:34.438801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:34.438794 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5" event={"ID":"9a87911d-6962-44f8-9307-5b8a70dab36c","Type":"ContainerDied","Data":"17abcc4ba17e1cfd75d0bee0ba60e599701c980e93706c0b03876e9039a95ba5"} Oct 29 23:29:34.439096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:34.438843 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17abcc4ba17e1cfd75d0bee0ba60e599701c980e93706c0b03876e9039a95ba5" Oct 29 23:29:35.346886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.346853 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt"] Oct 29 23:29:35.347736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.347714 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e811fbc8-69f4-4996-b14e-04aa58317174" containerName="payload-processing" Oct 29 23:29:35.347736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.347736 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e811fbc8-69f4-4996-b14e-04aa58317174" containerName="payload-processing" Oct 29 23:29:35.347850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.347778 5736 cpu_manager.go:414] "containerMap: removing container" podUID="34733fb2-3013-4eb8-9d44-5b3be1b3a263" containerName="payload-processing" Oct 29 23:29:35.347850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.347783 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34733fb2-3013-4eb8-9d44-5b3be1b3a263" containerName="payload-processing" Oct 29 23:29:35.347850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.347797 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9a87911d-6962-44f8-9307-5b8a70dab36c" containerName="revoke-keys" Oct 29 23:29:35.347850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.347802 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a87911d-6962-44f8-9307-5b8a70dab36c" containerName="revoke-keys" Oct 29 23:29:35.347973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.347918 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e811fbc8-69f4-4996-b14e-04aa58317174" containerName="payload-processing" Oct 29 23:29:35.347973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.347934 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="34733fb2-3013-4eb8-9d44-5b3be1b3a263" containerName="payload-processing" Oct 29 23:29:35.347973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.347942 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a87911d-6962-44f8-9307-5b8a70dab36c" containerName="revoke-keys" Oct 29 23:29:35.352110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.352083 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.384715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.384686 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt"] Oct 29 23:29:35.395342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.395309 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.395561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.395427 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.395561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.395451 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkh2z\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-kube-api-access-wkh2z\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.406624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.396654 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.499106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.499060 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.499301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.499217 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.499301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.499251 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkh2z\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-kube-api-access-wkh2z\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.499429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.499335 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.499971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.499944 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.500800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.500724 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.502818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.502788 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.549719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.539740 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkh2z\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-kube-api-access-wkh2z\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:35.668132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:35.668016 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:36.051185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.051155 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt"] Oct 29 23:29:36.054720 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:36.054685 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bbca4a_31cc_4533_adc2_4a29b7a5a2f3.slice/crio-a0ebe9373109ec211f52873e52a7ccc7a776afcf70078cf29ce0a0d4b1d1495a WatchSource:0}: Error finding container a0ebe9373109ec211f52873e52a7ccc7a776afcf70078cf29ce0a0d4b1d1495a: Status 404 returned error can't find the container with id a0ebe9373109ec211f52873e52a7ccc7a776afcf70078cf29ce0a0d4b1d1495a Oct 29 23:29:36.507225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.507185 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" event={"ID":"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3","Type":"ContainerStarted","Data":"be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86"} Oct 29 23:29:36.507820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.507789 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" event={"ID":"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3","Type":"ContainerStarted","Data":"a0ebe9373109ec211f52873e52a7ccc7a776afcf70078cf29ce0a0d4b1d1495a"} Oct 29 23:29:36.509121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.509099 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:36.578806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.578738 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" podStartSLOduration=1.578716065 podStartE2EDuration="1.578716065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:36.554205645 +0000 UTC m=+2773.670125253" watchObservedRunningTime="2026-10-29 23:29:36.578716065 +0000 UTC m=+2773.694635673" Oct 29 23:29:36.655876 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:36.654164 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dea9cb/agc-cred-189f55b8d790: secret "agc-cred-189f55b8d790" not found Oct 29 23:29:36.655876 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:36.654205 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt: secret "agc-cred-189f55b8d790" not found Oct 29 23:29:36.655876 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:36.654282 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials podName:94bbca4a-31cc-4533-adc2-4a29b7a5a2f3 nodeName:}" failed. No retries permitted until 2026-10-29 23:29:37.154261321 +0000 UTC m=+2774.270180926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" (UID: "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3") : secret "agc-cred-189f55b8d790" not found Oct 29 23:29:36.873679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.873598 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv"] Oct 29 23:29:36.878431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.878408 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:36.893196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.893174 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:36.893408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.893394 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:29:36.898264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.898238 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv"] Oct 29 23:29:36.911681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.911653 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt"] Oct 29 23:29:36.946211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.946181 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l"] Oct 29 23:29:36.950815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.950795 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:36.957957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.957936 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlcwf\" (UniqueName: \"kubernetes.io/projected/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-kube-api-access-jlcwf\") pod \"maas-api-revoke-sub-44a30064-29lbv\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:36.958206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.958185 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44a30064-29lbv\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:36.958396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.958358 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44a30064-29lbv\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:36.971087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:36.971064 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l"] Oct 29 23:29:37.059977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.059937 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.060162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.059993 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlcwf\" (UniqueName: \"kubernetes.io/projected/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-kube-api-access-jlcwf\") pod \"maas-api-revoke-sub-44a30064-29lbv\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:37.060162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.060135 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwlfn\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-kube-api-access-kwlfn\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.060343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.060324 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.060381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.060355 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.060424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.060406 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44a30064-29lbv\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:37.060703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.060674 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44a30064-29lbv\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:37.061077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.061050 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44a30064-29lbv\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:37.063015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.062998 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44a30064-29lbv\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:37.081779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.081759 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlcwf\" (UniqueName: \"kubernetes.io/projected/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-kube-api-access-jlcwf\") pod \"maas-api-revoke-sub-44a30064-29lbv\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:37.162195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.162126 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwlfn\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-kube-api-access-kwlfn\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.162321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.162260 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.162321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.162282 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.162467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.162448 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.162534 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:37.162491 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dea9cb/agc-cred-189f55b8d790: secret "agc-cred-189f55b8d790" not found Oct 29 23:29:37.162534 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:37.162517 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt: secret "agc-cred-189f55b8d790" not found Oct 29 23:29:37.162658 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:37.162597 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials podName:94bbca4a-31cc-4533-adc2-4a29b7a5a2f3 nodeName:}" failed. No retries permitted until 2026-10-29 23:29:38.16256353 +0000 UTC m=+2775.278483116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" (UID: "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3") : secret "agc-cred-189f55b8d790" not found Oct 29 23:29:37.162921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.162897 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.163135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.163116 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.164813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.164791 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.183339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.183318 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwlfn\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-kube-api-access-kwlfn\") pod \"payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.193225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.193203 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:37.265909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.265882 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:37.454843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.454768 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:29:37.511426 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:37.508000 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20283c0_98d0_4c3e_a5e1_8dcb47817b36.slice/crio-4e47917d3789594f9a10c99524348fcf9834ab57e0de2c30abe75763c30b852e WatchSource:0}: Error finding container 4e47917d3789594f9a10c99524348fcf9834ab57e0de2c30abe75763c30b852e: Status 404 returned error can't find the container with id 4e47917d3789594f9a10c99524348fcf9834ab57e0de2c30abe75763c30b852e Oct 29 23:29:37.516326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.516295 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv"] Oct 29 23:29:37.581228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:37.581199 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l"] Oct 29 23:29:37.586039 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:37.586010 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b3b3b0_b74c_4aac_b70b_b6431cbb042a.slice/crio-a102dd57aef4d2664b0f587c3f7a08c75a2095af7342a413f9139f4bf42c4daf WatchSource:0}: Error finding container a102dd57aef4d2664b0f587c3f7a08c75a2095af7342a413f9139f4bf42c4daf: Status 404 returned error can't find the container with id a102dd57aef4d2664b0f587c3f7a08c75a2095af7342a413f9139f4bf42c4daf Oct 29 23:29:38.209365 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:38.209333 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dea9cb/agc-cred-189f55b8d790: secret "agc-cred-189f55b8d790" not found Oct 29 23:29:38.209365 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:38.209365 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt: secret "agc-cred-189f55b8d790" not found Oct 29 23:29:38.209592 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:38.209418 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials podName:94bbca4a-31cc-4533-adc2-4a29b7a5a2f3 nodeName:}" failed. No retries permitted until 2026-10-29 23:29:40.209403778 +0000 UTC m=+2777.325323361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" (UID: "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3") : secret "agc-cred-189f55b8d790" not found Oct 29 23:29:38.529482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:38.529444 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" event={"ID":"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a","Type":"ContainerStarted","Data":"8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51"} Oct 29 23:29:38.529935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:38.529524 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" event={"ID":"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a","Type":"ContainerStarted","Data":"a102dd57aef4d2664b0f587c3f7a08c75a2095af7342a413f9139f4bf42c4daf"} Oct 29 23:29:38.529935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:38.529689 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:38.531375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:38.531346 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" event={"ID":"c20283c0-98d0-4c3e-a5e1-8dcb47817b36","Type":"ContainerStarted","Data":"a95ffafb294a9f01468bd91106314f78432912ca1e6e2550121061738c515a6e"} Oct 29 23:29:38.531526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:38.531399 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" event={"ID":"c20283c0-98d0-4c3e-a5e1-8dcb47817b36","Type":"ContainerStarted","Data":"4e47917d3789594f9a10c99524348fcf9834ab57e0de2c30abe75763c30b852e"} Oct 29 23:29:38.568313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:38.568252 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" podStartSLOduration=2.568232385 podStartE2EDuration="2.568232385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:38.567044678 +0000 UTC m=+2775.682964284" watchObservedRunningTime="2026-10-29 23:29:38.568232385 +0000 UTC m=+2775.684151992" Oct 29 23:29:38.646203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:38.646153 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" podStartSLOduration=2.646136067 podStartE2EDuration="2.646136067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:38.64256448 +0000 UTC m=+2775.758484100" watchObservedRunningTime="2026-10-29 23:29:38.646136067 +0000 UTC m=+2775.762055672" Oct 29 23:29:39.541014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:39.540977 5736 generic.go:356] "Generic (PLEG): container finished" podID="c20283c0-98d0-4c3e-a5e1-8dcb47817b36" containerID="a95ffafb294a9f01468bd91106314f78432912ca1e6e2550121061738c515a6e" exitCode=0 Oct 29 23:29:39.541413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:39.541059 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" event={"ID":"c20283c0-98d0-4c3e-a5e1-8dcb47817b36","Type":"ContainerDied","Data":"a95ffafb294a9f01468bd91106314f78432912ca1e6e2550121061738c515a6e"} Oct 29 23:29:40.254193 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:40.254161 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dea9cb/agc-cred-189f55b8d790: secret "agc-cred-189f55b8d790" not found Oct 29 23:29:40.254193 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:40.254191 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt: secret "agc-cred-189f55b8d790" not found Oct 29 23:29:40.254455 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:40.254254 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials podName:94bbca4a-31cc-4533-adc2-4a29b7a5a2f3 nodeName:}" failed. No retries permitted until 2026-10-29 23:29:44.254231658 +0000 UTC m=+2781.370151267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" (UID: "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3") : secret "agc-cred-189f55b8d790" not found Oct 29 23:29:40.724839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.724809 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:40.862594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.862485 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-xks-ca\") pod \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " Oct 29 23:29:40.862819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.862800 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-kube-api-access-jlcwf\" (UniqueName: \"kubernetes.io/projected/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-kube-api-access-jlcwf\") pod \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " Oct 29 23:29:40.862948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.862930 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-openshift-ca\") pod \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\" (UID: \"c20283c0-98d0-4c3e-a5e1-8dcb47817b36\") " Oct 29 23:29:40.863348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.863321 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-openshift-ca" pod "c20283c0-98d0-4c3e-a5e1-8dcb47817b36" (UID: "c20283c0-98d0-4c3e-a5e1-8dcb47817b36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:40.863767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.863742 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:40.864761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.864738 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-xks-ca" pod "c20283c0-98d0-4c3e-a5e1-8dcb47817b36" (UID: "c20283c0-98d0-4c3e-a5e1-8dcb47817b36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:40.865018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.865001 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-kube-api-access-jlcwf" pod "c20283c0-98d0-4c3e-a5e1-8dcb47817b36" (UID: "c20283c0-98d0-4c3e-a5e1-8dcb47817b36"). InnerVolumeSpecName "kube-api-access-jlcwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:40.965363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.965331 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:40.965363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:40.965361 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlcwf\" (UniqueName: \"kubernetes.io/projected/c20283c0-98d0-4c3e-a5e1-8dcb47817b36-kube-api-access-jlcwf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:41.461069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:41.460246 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn"] Oct 29 23:29:41.565563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:41.565522 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" Oct 29 23:29:41.566422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:41.565537 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv" event={"ID":"c20283c0-98d0-4c3e-a5e1-8dcb47817b36","Type":"ContainerDied","Data":"4e47917d3789594f9a10c99524348fcf9834ab57e0de2c30abe75763c30b852e"} Oct 29 23:29:41.566555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:41.566454 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e47917d3789594f9a10c99524348fcf9834ab57e0de2c30abe75763c30b852e" Oct 29 23:29:43.532467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.532429 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" podUID="94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" containerName="payload-processing" containerID="cri-o://be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86" gracePeriod=25 Oct 29 23:29:43.820716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.820684 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:43.962226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.962188 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials\") pod \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " Oct 29 23:29:43.962420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.962302 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-kube-api-access-wkh2z\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-kube-api-access-wkh2z\") pod \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " Oct 29 23:29:43.962420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.962326 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-plugins-config-volume\") pod \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " Oct 29 23:29:43.962420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.962360 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-routing-overlay-volume\") pod \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\" (UID: \"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3\") " Oct 29 23:29:43.963103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.963069 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-plugins-config-volume" pod "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" (UID: "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:43.963274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.963119 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-routing-overlay-volume" pod "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" (UID: "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:43.964573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.964548 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials" pod "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" (UID: "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:43.964716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:43.964604 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-kube-api-access-wkh2z" pod "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" (UID: "94bbca4a-31cc-4533-adc2-4a29b7a5a2f3"). InnerVolumeSpecName "kube-api-access-wkh2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:44.065294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.065199 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:44.065294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.065237 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkh2z\" (UniqueName: \"kubernetes.io/projected/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-kube-api-access-wkh2z\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:44.065294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.065247 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:44.065294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.065256 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:44.596648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.596573 5736 generic.go:356] "Generic (PLEG): container finished" podID="94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" containerID="be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86" exitCode=0 Oct 29 23:29:44.597076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.596661 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" Oct 29 23:29:44.597076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.596691 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" event={"ID":"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3","Type":"ContainerDied","Data":"be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86"} Oct 29 23:29:44.597076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.596750 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt" event={"ID":"94bbca4a-31cc-4533-adc2-4a29b7a5a2f3","Type":"ContainerDied","Data":"a0ebe9373109ec211f52873e52a7ccc7a776afcf70078cf29ce0a0d4b1d1495a"} Oct 29 23:29:44.597076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.596781 5736 scope.go:122] "RemoveContainer" containerID="be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86" Oct 29 23:29:44.608616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.608561 5736 scope.go:122] "RemoveContainer" containerID="be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86" Oct 29 23:29:44.608872 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:44.608852 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86\": container with ID starting with be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86 not found: ID does not exist" containerID="be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86" Oct 29 23:29:44.608924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.608878 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86"} err="failed to get container status \"be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86\": rpc error: code = NotFound desc = could not find container \"be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86\": container with ID starting with be1fdec6b49db92422964077d1e10c8b3197477f0832e77659b7054b2282af86 not found: ID does not exist" Oct 29 23:29:44.644210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.644183 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt"] Oct 29 23:29:44.654487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.654462 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-5fc979rwt"] Oct 29 23:29:44.702045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.701983 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p"] Oct 29 23:29:44.703172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.703150 5736 cpu_manager.go:414] "containerMap: removing container" podUID="94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" containerName="payload-processing" Oct 29 23:29:44.703246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.703174 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" containerName="payload-processing" Oct 29 23:29:44.703246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.703192 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c20283c0-98d0-4c3e-a5e1-8dcb47817b36" containerName="revoke-keys" Oct 29 23:29:44.703246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.703203 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20283c0-98d0-4c3e-a5e1-8dcb47817b36" containerName="revoke-keys" Oct 29 23:29:44.703501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.703485 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" containerName="payload-processing" Oct 29 23:29:44.703544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.703508 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20283c0-98d0-4c3e-a5e1-8dcb47817b36" containerName="revoke-keys" Oct 29 23:29:44.709213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.709193 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:44.722383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.722355 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:44.722500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.722367 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:29:44.725445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.725419 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p"] Oct 29 23:29:44.877209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.877119 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ac438a-qz27p\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:44.877209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.877185 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvbd\" (UniqueName: \"kubernetes.io/projected/b873c346-5539-42ad-926e-a92c37559eb9-kube-api-access-fcvbd\") pod \"maas-api-revoke-sub-e1ac438a-qz27p\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:44.877399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.877336 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ac438a-qz27p\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:44.978891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.978859 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ac438a-qz27p\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:44.979069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.978912 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvbd\" (UniqueName: \"kubernetes.io/projected/b873c346-5539-42ad-926e-a92c37559eb9-kube-api-access-fcvbd\") pod \"maas-api-revoke-sub-e1ac438a-qz27p\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:44.979069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.979005 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ac438a-qz27p\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:44.979543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.979517 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ac438a-qz27p\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:44.981563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:44.981538 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ac438a-qz27p\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:45.001350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:45.001323 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvbd\" (UniqueName: \"kubernetes.io/projected/b873c346-5539-42ad-926e-a92c37559eb9-kube-api-access-fcvbd\") pod \"maas-api-revoke-sub-e1ac438a-qz27p\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:45.009182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:45.009145 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94bbca4a-31cc-4533-adc2-4a29b7a5a2f3" path="/var/lib/kubelet/pods/94bbca4a-31cc-4533-adc2-4a29b7a5a2f3/volumes" Oct 29 23:29:45.027438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:45.027413 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:45.338794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:45.338764 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p"] Oct 29 23:29:45.340753 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:45.340709 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb873c346_5539_42ad_926e_a92c37559eb9.slice/crio-991fc82c37f6aee0cba26bf62da172b975b1940c71032932fc4cb4ab4eed37ce WatchSource:0}: Error finding container 991fc82c37f6aee0cba26bf62da172b975b1940c71032932fc4cb4ab4eed37ce: Status 404 returned error can't find the container with id 991fc82c37f6aee0cba26bf62da172b975b1940c71032932fc4cb4ab4eed37ce Oct 29 23:29:45.608748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:45.608667 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" event={"ID":"b873c346-5539-42ad-926e-a92c37559eb9","Type":"ContainerStarted","Data":"991fc82c37f6aee0cba26bf62da172b975b1940c71032932fc4cb4ab4eed37ce"} Oct 29 23:29:46.460962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.460927 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" podUID="bfe388bf-3ac9-40c1-9559-ba3b042df7c7" containerName="payload-processing" containerID="cri-o://2b04ddc58be1307a7b345e5ecb4dafe6b8d7190e6049a252eaf6037493b92329" gracePeriod=25 Oct 29 23:29:46.619159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.619079 5736 generic.go:356] "Generic (PLEG): container finished" podID="bfe388bf-3ac9-40c1-9559-ba3b042df7c7" containerID="2b04ddc58be1307a7b345e5ecb4dafe6b8d7190e6049a252eaf6037493b92329" exitCode=0 Oct 29 23:29:46.619544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.619265 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" event={"ID":"bfe388bf-3ac9-40c1-9559-ba3b042df7c7","Type":"ContainerDied","Data":"2b04ddc58be1307a7b345e5ecb4dafe6b8d7190e6049a252eaf6037493b92329"} Oct 29 23:29:46.621301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.621270 5736 generic.go:356] "Generic (PLEG): container finished" podID="b873c346-5539-42ad-926e-a92c37559eb9" containerID="16719025f5649296312b49315ef438bf588f46d81fe34061ae116f7e16f2b0ee" exitCode=0 Oct 29 23:29:46.621432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.621349 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" event={"ID":"b873c346-5539-42ad-926e-a92c37559eb9","Type":"ContainerDied","Data":"16719025f5649296312b49315ef438bf588f46d81fe34061ae116f7e16f2b0ee"} Oct 29 23:29:46.737006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.736980 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:29:46.820089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.820056 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-provider-credentials\") pod \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " Oct 29 23:29:46.820248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.820201 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-routing-overlay-volume\") pod \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " Oct 29 23:29:46.820338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.820287 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-plugins-config-volume\") pod \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " Oct 29 23:29:46.820595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.820559 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-kube-api-access-qr6b5\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-kube-api-access-qr6b5\") pod \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\" (UID: \"bfe388bf-3ac9-40c1-9559-ba3b042df7c7\") " Oct 29 23:29:46.820745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.820638 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-routing-overlay-volume" pod "bfe388bf-3ac9-40c1-9559-ba3b042df7c7" (UID: "bfe388bf-3ac9-40c1-9559-ba3b042df7c7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:46.820817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.820765 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-plugins-config-volume" pod "bfe388bf-3ac9-40c1-9559-ba3b042df7c7" (UID: "bfe388bf-3ac9-40c1-9559-ba3b042df7c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:46.821366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.821340 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.821474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.821370 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.822884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.822852 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-kube-api-access-qr6b5" pod "bfe388bf-3ac9-40c1-9559-ba3b042df7c7" (UID: "bfe388bf-3ac9-40c1-9559-ba3b042df7c7"). InnerVolumeSpecName "kube-api-access-qr6b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:46.823009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.822987 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-provider-credentials" pod "bfe388bf-3ac9-40c1-9559-ba3b042df7c7" (UID: "bfe388bf-3ac9-40c1-9559-ba3b042df7c7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:46.922489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.922421 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr6b5\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-kube-api-access-qr6b5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.922489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:46.922449 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfe388bf-3ac9-40c1-9559-ba3b042df7c7-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:47.679442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.679402 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" Oct 29 23:29:47.679442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.679426 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn" event={"ID":"bfe388bf-3ac9-40c1-9559-ba3b042df7c7","Type":"ContainerDied","Data":"598fb027b2dd513be94110e14559286287e591373e24d1849940e3567916f750"} Oct 29 23:29:47.680046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.679494 5736 scope.go:122] "RemoveContainer" containerID="2b04ddc58be1307a7b345e5ecb4dafe6b8d7190e6049a252eaf6037493b92329" Oct 29 23:29:47.728288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.728187 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn"] Oct 29 23:29:47.738160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.738129 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-c7b8g4qdn"] Oct 29 23:29:47.846156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.846134 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:47.939215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.939140 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-xks-ca\") pod \"b873c346-5539-42ad-926e-a92c37559eb9\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " Oct 29 23:29:47.939215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.939197 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-openshift-ca\") pod \"b873c346-5539-42ad-926e-a92c37559eb9\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " Oct 29 23:29:47.939452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.939401 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b873c346-5539-42ad-926e-a92c37559eb9-kube-api-access-fcvbd\" (UniqueName: \"kubernetes.io/projected/b873c346-5539-42ad-926e-a92c37559eb9-kube-api-access-fcvbd\") pod \"b873c346-5539-42ad-926e-a92c37559eb9\" (UID: \"b873c346-5539-42ad-926e-a92c37559eb9\") " Oct 29 23:29:47.939648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.939622 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-openshift-ca" pod "b873c346-5539-42ad-926e-a92c37559eb9" (UID: "b873c346-5539-42ad-926e-a92c37559eb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:47.940060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.940041 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:47.941275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.941251 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-xks-ca" pod "b873c346-5539-42ad-926e-a92c37559eb9" (UID: "b873c346-5539-42ad-926e-a92c37559eb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:47.941379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:47.941360 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b873c346-5539-42ad-926e-a92c37559eb9-kube-api-access-fcvbd" pod "b873c346-5539-42ad-926e-a92c37559eb9" (UID: "b873c346-5539-42ad-926e-a92c37559eb9"). InnerVolumeSpecName "kube-api-access-fcvbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:48.041451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:48.041417 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcvbd\" (UniqueName: \"kubernetes.io/projected/b873c346-5539-42ad-926e-a92c37559eb9-kube-api-access-fcvbd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:48.041451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:48.041447 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b873c346-5539-42ad-926e-a92c37559eb9-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:48.690074 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:48.690046 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" Oct 29 23:29:48.690488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:48.690090 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p" event={"ID":"b873c346-5539-42ad-926e-a92c37559eb9","Type":"ContainerDied","Data":"991fc82c37f6aee0cba26bf62da172b975b1940c71032932fc4cb4ab4eed37ce"} Oct 29 23:29:48.690488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:48.690147 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="991fc82c37f6aee0cba26bf62da172b975b1940c71032932fc4cb4ab4eed37ce" Oct 29 23:29:49.005377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:49.005335 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe388bf-3ac9-40c1-9559-ba3b042df7c7" path="/var/lib/kubelet/pods/bfe388bf-3ac9-40c1-9559-ba3b042df7c7/volumes" Oct 29 23:29:49.231866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:49.231821 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs"] Oct 29 23:29:49.239163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:49.238805 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l"] Oct 29 23:29:49.250591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:49.250543 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:50.685502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.685467 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847"] Oct 29 23:29:50.686468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.686442 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bfe388bf-3ac9-40c1-9559-ba3b042df7c7" containerName="payload-processing" Oct 29 23:29:50.686603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.686468 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe388bf-3ac9-40c1-9559-ba3b042df7c7" containerName="payload-processing" Oct 29 23:29:50.686603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.686556 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b873c346-5539-42ad-926e-a92c37559eb9" containerName="revoke-keys" Oct 29 23:29:50.686603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.686566 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b873c346-5539-42ad-926e-a92c37559eb9" containerName="revoke-keys" Oct 29 23:29:50.686791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.686775 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b873c346-5539-42ad-926e-a92c37559eb9" containerName="revoke-keys" Oct 29 23:29:50.686855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.686800 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe388bf-3ac9-40c1-9559-ba3b042df7c7" containerName="payload-processing" Oct 29 23:29:50.690859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.690839 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:50.704718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.704690 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:50.709518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.709488 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847"] Oct 29 23:29:50.714984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.714938 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:29:50.804268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.804234 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltkk\" (UniqueName: \"kubernetes.io/projected/693b6d84-ab38-4cc7-a9ec-55184c781679-kube-api-access-4ltkk\") pod \"maas-api-revoke-sub-73ffa982-hb847\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:50.804531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.804508 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ffa982-hb847\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:50.804596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.804554 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ffa982-hb847\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:50.905874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.905836 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ffa982-hb847\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:50.906056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.905890 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ffa982-hb847\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:50.906056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.906011 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltkk\" (UniqueName: \"kubernetes.io/projected/693b6d84-ab38-4cc7-a9ec-55184c781679-kube-api-access-4ltkk\") pod \"maas-api-revoke-sub-73ffa982-hb847\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:50.906625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.906603 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ffa982-hb847\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:50.908212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.908191 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ffa982-hb847\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:50.927433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:50.927403 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltkk\" (UniqueName: \"kubernetes.io/projected/693b6d84-ab38-4cc7-a9ec-55184c781679-kube-api-access-4ltkk\") pod \"maas-api-revoke-sub-73ffa982-hb847\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:51.004538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:51.004511 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:51.314848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:51.314820 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847"] Oct 29 23:29:51.316762 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:29:51.316731 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693b6d84_ab38_4cc7_a9ec_55184c781679.slice/crio-dc25dfb9ae11f2dffcd684e8c548eef00c6725dbf6274a14d972cc8c73b3e198 WatchSource:0}: Error finding container dc25dfb9ae11f2dffcd684e8c548eef00c6725dbf6274a14d972cc8c73b3e198: Status 404 returned error can't find the container with id dc25dfb9ae11f2dffcd684e8c548eef00c6725dbf6274a14d972cc8c73b3e198 Oct 29 23:29:51.741510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:51.741480 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" event={"ID":"693b6d84-ab38-4cc7-a9ec-55184c781679","Type":"ContainerStarted","Data":"dc25dfb9ae11f2dffcd684e8c548eef00c6725dbf6274a14d972cc8c73b3e198"} Oct 29 23:29:52.751363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:52.751319 5736 generic.go:356] "Generic (PLEG): container finished" podID="693b6d84-ab38-4cc7-a9ec-55184c781679" containerID="ea83a8df00534b9f56bef7cbc5211c6bd135c685d46f20e583fc8d05d266aa70" exitCode=0 Oct 29 23:29:52.751785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:52.751410 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" event={"ID":"693b6d84-ab38-4cc7-a9ec-55184c781679","Type":"ContainerDied","Data":"ea83a8df00534b9f56bef7cbc5211c6bd135c685d46f20e583fc8d05d266aa70"} Oct 29 23:29:53.958792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:53.958765 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:54.054933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.054861 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-openshift-ca\") pod \"693b6d84-ab38-4cc7-a9ec-55184c781679\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " Oct 29 23:29:54.055106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.055089 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-xks-ca\") pod \"693b6d84-ab38-4cc7-a9ec-55184c781679\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " Oct 29 23:29:54.055202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.055187 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693b6d84-ab38-4cc7-a9ec-55184c781679-kube-api-access-4ltkk\" (UniqueName: \"kubernetes.io/projected/693b6d84-ab38-4cc7-a9ec-55184c781679-kube-api-access-4ltkk\") pod \"693b6d84-ab38-4cc7-a9ec-55184c781679\" (UID: \"693b6d84-ab38-4cc7-a9ec-55184c781679\") " Oct 29 23:29:54.055350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.055318 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-openshift-ca" pod "693b6d84-ab38-4cc7-a9ec-55184c781679" (UID: "693b6d84-ab38-4cc7-a9ec-55184c781679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:54.056057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.056029 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.057462 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.057424 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693b6d84-ab38-4cc7-a9ec-55184c781679-kube-api-access-4ltkk" pod "693b6d84-ab38-4cc7-a9ec-55184c781679" (UID: "693b6d84-ab38-4cc7-a9ec-55184c781679"). InnerVolumeSpecName "kube-api-access-4ltkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:54.059703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.059679 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-xks-ca" pod "693b6d84-ab38-4cc7-a9ec-55184c781679" (UID: "693b6d84-ab38-4cc7-a9ec-55184c781679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:54.157301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.157274 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693b6d84-ab38-4cc7-a9ec-55184c781679-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.157416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.157302 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ltkk\" (UniqueName: \"kubernetes.io/projected/693b6d84-ab38-4cc7-a9ec-55184c781679-kube-api-access-4ltkk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.233397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.233323 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" podUID="58f7b37e-e93a-4ab5-8f18-bd194b8f966d" containerName="payload-processing" containerID="cri-o://342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326" gracePeriod=25 Oct 29 23:29:54.240067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.240040 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" podUID="d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" containerName="payload-processing" containerID="cri-o://8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51" gracePeriod=25 Oct 29 23:29:54.508073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.508047 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:29:54.549997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.549967 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:54.664900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.664825 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-plugins-config-volume\") pod \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " Oct 29 23:29:54.664900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.664870 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-routing-overlay-volume\") pod \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " Oct 29 23:29:54.665102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.664956 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-plugins-config-volume\") pod \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " Oct 29 23:29:54.665102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.665041 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-provider-credentials\") pod \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " Oct 29 23:29:54.665232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.665138 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-kube-api-access-zkgl5\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-kube-api-access-zkgl5\") pod \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\" (UID: \"58f7b37e-e93a-4ab5-8f18-bd194b8f966d\") " Oct 29 23:29:54.665232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.665201 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-kube-api-access-kwlfn\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-kube-api-access-kwlfn\") pod \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " Oct 29 23:29:54.665350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.665257 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-provider-credentials\") pod \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " Oct 29 23:29:54.665350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.665317 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-plugins-config-volume" pod "58f7b37e-e93a-4ab5-8f18-bd194b8f966d" (UID: "58f7b37e-e93a-4ab5-8f18-bd194b8f966d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:54.665350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.665327 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-routing-overlay-volume" pod "58f7b37e-e93a-4ab5-8f18-bd194b8f966d" (UID: "58f7b37e-e93a-4ab5-8f18-bd194b8f966d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:54.665350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.665345 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-routing-overlay-volume\") pod \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\" (UID: \"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a\") " Oct 29 23:29:54.665593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.665428 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-plugins-config-volume" pod "d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" (UID: "d6b3b3b0-b74c-4aac-b70b-b6431cbb042a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:54.665930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.665908 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-routing-overlay-volume" pod "d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" (UID: "d6b3b3b0-b74c-4aac-b70b-b6431cbb042a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:54.666112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.666094 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.666200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.666115 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.666200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.666130 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.666200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.666146 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.667621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.667573 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-kube-api-access-zkgl5" pod "58f7b37e-e93a-4ab5-8f18-bd194b8f966d" (UID: "58f7b37e-e93a-4ab5-8f18-bd194b8f966d"). InnerVolumeSpecName "kube-api-access-zkgl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:54.667621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.667621 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-provider-credentials" pod "58f7b37e-e93a-4ab5-8f18-bd194b8f966d" (UID: "58f7b37e-e93a-4ab5-8f18-bd194b8f966d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:54.667934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.667909 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-kube-api-access-kwlfn" pod "d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" (UID: "d6b3b3b0-b74c-4aac-b70b-b6431cbb042a"). InnerVolumeSpecName "kube-api-access-kwlfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:54.668032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.667934 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-provider-credentials" pod "d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" (UID: "d6b3b3b0-b74c-4aac-b70b-b6431cbb042a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:54.767285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.767252 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.767285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.767286 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkgl5\" (UniqueName: \"kubernetes.io/projected/58f7b37e-e93a-4ab5-8f18-bd194b8f966d-kube-api-access-zkgl5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.767465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.767301 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwlfn\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-kube-api-access-kwlfn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.767465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.767316 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:54.802958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.802924 5736 generic.go:356] "Generic (PLEG): container finished" podID="d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" containerID="8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51" exitCode=0 Oct 29 23:29:54.803107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.802959 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" Oct 29 23:29:54.803107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.803049 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" event={"ID":"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a","Type":"ContainerDied","Data":"8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51"} Oct 29 23:29:54.803107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.803105 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l" event={"ID":"d6b3b3b0-b74c-4aac-b70b-b6431cbb042a","Type":"ContainerDied","Data":"a102dd57aef4d2664b0f587c3f7a08c75a2095af7342a413f9139f4bf42c4daf"} Oct 29 23:29:54.803293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.803134 5736 scope.go:122] "RemoveContainer" containerID="8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51" Oct 29 23:29:54.804836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.804809 5736 generic.go:356] "Generic (PLEG): container finished" podID="58f7b37e-e93a-4ab5-8f18-bd194b8f966d" containerID="342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326" exitCode=0 Oct 29 23:29:54.804961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.804855 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" Oct 29 23:29:54.804961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.804899 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" event={"ID":"58f7b37e-e93a-4ab5-8f18-bd194b8f966d","Type":"ContainerDied","Data":"342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326"} Oct 29 23:29:54.804961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.804950 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs" event={"ID":"58f7b37e-e93a-4ab5-8f18-bd194b8f966d","Type":"ContainerDied","Data":"6d82f1c24b6e44ee6f6898c7697c57b67a0623f84939021edcc1bdc45fdf4ef4"} Oct 29 23:29:54.814543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.814524 5736 scope.go:122] "RemoveContainer" containerID="8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51" Oct 29 23:29:54.815069 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:54.815024 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51\": container with ID starting with 8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51 not found: ID does not exist" containerID="8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51" Oct 29 23:29:54.815178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.815080 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" event={"ID":"693b6d84-ab38-4cc7-a9ec-55184c781679","Type":"ContainerDied","Data":"dc25dfb9ae11f2dffcd684e8c548eef00c6725dbf6274a14d972cc8c73b3e198"} Oct 29 23:29:54.815178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.815076 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51"} err="failed to get container status \"8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51\": rpc error: code = NotFound desc = could not find container \"8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51\": container with ID starting with 8806346ac4115f6eacfdb9d00f72b27144f47cc6c02981d37496cedcaff4dd51 not found: ID does not exist" Oct 29 23:29:54.815178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.815103 5736 scope.go:122] "RemoveContainer" containerID="342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326" Oct 29 23:29:54.815178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.815120 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc25dfb9ae11f2dffcd684e8c548eef00c6725dbf6274a14d972cc8c73b3e198" Oct 29 23:29:54.815178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.815048 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847" Oct 29 23:29:54.827312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.827295 5736 scope.go:122] "RemoveContainer" containerID="342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326" Oct 29 23:29:54.827684 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:54.827655 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326\": container with ID starting with 342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326 not found: ID does not exist" containerID="342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326" Oct 29 23:29:54.827751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.827696 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326"} err="failed to get container status \"342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326\": rpc error: code = NotFound desc = could not find container \"342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326\": container with ID starting with 342639760fba9afa323737ca78bcdcd893fd187d3fd2775904463ef781c3d326 not found: ID does not exist" Oct 29 23:29:54.865684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.865658 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs"] Oct 29 23:29:54.869937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.869918 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-54b966hhs"] Oct 29 23:29:54.906159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.906136 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l"] Oct 29 23:29:54.913136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:54.913115 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dea9cb/payload-processing-external-model-e2e-extmod-a-dea9cb-7d47sx92l"] Oct 29 23:29:55.006273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:55.006245 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f7b37e-e93a-4ab5-8f18-bd194b8f966d" path="/var/lib/kubelet/pods/58f7b37e-e93a-4ab5-8f18-bd194b8f966d/volumes" Oct 29 23:29:55.006692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:55.006676 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" path="/var/lib/kubelet/pods/d6b3b3b0-b74c-4aac-b70b-b6431cbb042a/volumes" Oct 29 23:29:56.078146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:56.078110 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst"] Oct 29 23:29:57.010605 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:57.006042 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:29:57.562606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.556607 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv"] Oct 29 23:29:57.586604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.571323 5736 cpu_manager.go:414] "containerMap: removing container" podUID="58f7b37e-e93a-4ab5-8f18-bd194b8f966d" containerName="payload-processing" Oct 29 23:29:57.586604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.571355 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f7b37e-e93a-4ab5-8f18-bd194b8f966d" containerName="payload-processing" Oct 29 23:29:57.586604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.571377 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" containerName="payload-processing" Oct 29 23:29:57.586604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.571386 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" containerName="payload-processing" Oct 29 23:29:57.586604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.571406 5736 cpu_manager.go:414] "containerMap: removing container" podUID="693b6d84-ab38-4cc7-a9ec-55184c781679" containerName="revoke-keys" Oct 29 23:29:57.586604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.571413 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693b6d84-ab38-4cc7-a9ec-55184c781679" containerName="revoke-keys" Oct 29 23:29:57.586604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.571749 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f7b37e-e93a-4ab5-8f18-bd194b8f966d" containerName="payload-processing" Oct 29 23:29:57.586604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.571774 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b3b3b0-b74c-4aac-b70b-b6431cbb042a" containerName="payload-processing" Oct 29 23:29:57.586604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.571789 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="693b6d84-ab38-4cc7-a9ec-55184c781679" containerName="revoke-keys" Oct 29 23:29:57.591614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.588259 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.651601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.642920 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"payload-processing-external-model-e2e--ca6efe92-dockercfg-t7z26\"" Oct 29 23:29:57.651601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.643619 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"agc-cred-fd8530b2a258\"" Oct 29 23:29:57.653525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.653387 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.653525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.653449 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8nz4\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-kube-api-access-m8nz4\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.653525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.653487 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.653849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.653679 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.667327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.667298 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv"] Oct 29 23:29:57.755429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.755381 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.757339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.756017 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nz4\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-kube-api-access-m8nz4\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.757339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.756075 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.757339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.756149 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.757339 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:57.756481 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c5ef2f/agc-cred-fd8530b2a258: secret "agc-cred-fd8530b2a258" not found Oct 29 23:29:57.757339 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:57.756501 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv: secret "agc-cred-fd8530b2a258" not found Oct 29 23:29:57.757339 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:57.756614 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials podName:cd129ecb-6995-4dfc-bda1-4da0b20a5a79 nodeName:}" failed. No retries permitted until 2026-10-29 23:29:58.256554797 +0000 UTC m=+2795.372474388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" (UID: "cd129ecb-6995-4dfc-bda1-4da0b20a5a79") : secret "agc-cred-fd8530b2a258" not found Oct 29 23:29:57.757339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.756620 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.757840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.757408 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.832528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.820445 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nz4\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-kube-api-access-m8nz4\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:57.954602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:57.953737 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv"] Oct 29 23:29:57.954813 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:57.954717 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" podUID="cd129ecb-6995-4dfc-bda1-4da0b20a5a79" Oct 29 23:29:58.261545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.261507 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:58.261719 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:58.261702 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c5ef2f/agc-cred-fd8530b2a258: secret "agc-cred-fd8530b2a258" not found Oct 29 23:29:58.261777 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:58.261721 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv: secret "agc-cred-fd8530b2a258" not found Oct 29 23:29:58.261816 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:58.261778 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials podName:cd129ecb-6995-4dfc-bda1-4da0b20a5a79 nodeName:}" failed. No retries permitted until 2026-10-29 23:29:59.261761016 +0000 UTC m=+2796.377680620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" (UID: "cd129ecb-6995-4dfc-bda1-4da0b20a5a79") : secret "agc-cred-fd8530b2a258" not found Oct 29 23:29:58.921431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.921399 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:58.930392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.930365 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:58.979378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.979351 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-routing-overlay-volume\") pod \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " Oct 29 23:29:58.979534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.979397 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-plugins-config-volume\") pod \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " Oct 29 23:29:58.979626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.979614 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-kube-api-access-m8nz4\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-kube-api-access-m8nz4\") pod \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " Oct 29 23:29:58.979763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.979738 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-routing-overlay-volume" pod "cd129ecb-6995-4dfc-bda1-4da0b20a5a79" (UID: "cd129ecb-6995-4dfc-bda1-4da0b20a5a79"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:58.979913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.979889 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-plugins-config-volume" pod "cd129ecb-6995-4dfc-bda1-4da0b20a5a79" (UID: "cd129ecb-6995-4dfc-bda1-4da0b20a5a79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:58.980198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.980180 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.980270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.980203 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.981861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:58.981834 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-kube-api-access-m8nz4" pod "cd129ecb-6995-4dfc-bda1-4da0b20a5a79" (UID: "cd129ecb-6995-4dfc-bda1-4da0b20a5a79"). InnerVolumeSpecName "kube-api-access-m8nz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:59.081889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:59.081855 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8nz4\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-kube-api-access-m8nz4\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.284143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:59.284112 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv\" (UID: \"cd129ecb-6995-4dfc-bda1-4da0b20a5a79\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:29:59.284321 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:59.284262 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c5ef2f/agc-cred-fd8530b2a258: secret "agc-cred-fd8530b2a258" not found Oct 29 23:29:59.284321 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:59.284284 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv: secret "agc-cred-fd8530b2a258" not found Oct 29 23:29:59.284408 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:29:59.284339 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials podName:cd129ecb-6995-4dfc-bda1-4da0b20a5a79 nodeName:}" failed. No retries permitted until 2026-10-29 23:30:01.28432518 +0000 UTC m=+2798.400244768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" (UID: "cd129ecb-6995-4dfc-bda1-4da0b20a5a79") : secret "agc-cred-fd8530b2a258" not found Oct 29 23:29:59.928710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:29:59.928672 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv" Oct 29 23:30:00.002684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.002646 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv"] Oct 29 23:30:00.007809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.007780 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf62bnmv"] Oct 29 23:30:00.096250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.096209 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd129ecb-6995-4dfc-bda1-4da0b20a5a79-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:00.143693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.143612 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv"] Oct 29 23:30:00.149676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.149650 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:00.155360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.155330 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj"] Oct 29 23:30:00.160426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.160408 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb"] Oct 29 23:30:00.160557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.160533 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:00.165499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.165480 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2"] Oct 29 23:30:00.165629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.165613 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:00.170549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.170499 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2"] Oct 29 23:30:00.171267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.171244 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:00.176115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.176095 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv"] Oct 29 23:30:00.176264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.176247 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:00.182900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.182453 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n"] Oct 29 23:30:00.196245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.190023 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:00.196245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.195131 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:30:00.196245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.196236 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7"] Oct 29 23:30:00.210018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.209979 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8868c3b-f832-4b4c-8968-139096f19a4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n\" (UID: \"b8868c3b-f832-4b4c-8968-139096f19a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:00.210154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210025 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjdz\" (UniqueName: \"kubernetes.io/projected/08d3c108-ba23-4291-a0cf-909c58d2a8bb-kube-api-access-tmjdz\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj\" (UID: \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:00.210154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210057 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p42v\" (UniqueName: \"kubernetes.io/projected/c19f4669-9c01-4e64-a396-b755fdbfddd6-kube-api-access-4p42v\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2\" (UID: \"c19f4669-9c01-4e64-a396-b755fdbfddd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:00.210154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210101 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03a3d3-acce-41e7-97ff-47c0beb3a320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv\" (UID: \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:00.210154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210129 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f76f9f99-0971-49bb-b819-23ce4f897001-kube-api-access-2x4lx\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb\" (UID: \"f76f9f99-0971-49bb-b819-23ce4f897001\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:00.210404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210213 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhr5w\" (UniqueName: \"kubernetes.io/projected/8d03a3d3-acce-41e7-97ff-47c0beb3a320-kube-api-access-lhr5w\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv\" (UID: \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:00.210404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210280 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njftq\" (UniqueName: \"kubernetes.io/projected/992c6925-45f5-4cb3-a436-f032da9b30f5-kube-api-access-njftq\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2\" (UID: \"992c6925-45f5-4cb3-a436-f032da9b30f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:00.210404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210385 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d3c108-ba23-4291-a0cf-909c58d2a8bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj\" (UID: \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:00.210594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210421 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwtw\" (UniqueName: \"kubernetes.io/projected/b8868c3b-f832-4b4c-8968-139096f19a4a-kube-api-access-8vwtw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n\" (UID: \"b8868c3b-f832-4b4c-8968-139096f19a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:00.210594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210505 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19f4669-9c01-4e64-a396-b755fdbfddd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2\" (UID: \"c19f4669-9c01-4e64-a396-b755fdbfddd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:00.210710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210602 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c6925-45f5-4cb3-a436-f032da9b30f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2\" (UID: \"992c6925-45f5-4cb3-a436-f032da9b30f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:00.210710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.210687 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76f9f99-0971-49bb-b819-23ce4f897001-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb\" (UID: \"f76f9f99-0971-49bb-b819-23ce4f897001\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:00.211179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.211153 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7"] Oct 29 23:30:00.213859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.213829 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:00.228607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.227652 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj"] Oct 29 23:30:00.228607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.227700 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb"] Oct 29 23:30:00.228607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.227791 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:00.234975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.234469 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2"] Oct 29 23:30:00.241537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.241513 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2"] Oct 29 23:30:00.259806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.257626 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7"] Oct 29 23:30:00.260308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.260281 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb"] Oct 29 23:30:00.273635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.272768 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx"] Oct 29 23:30:00.277223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.272868 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:00.294402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.292774 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7"] Oct 29 23:30:00.294402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.292822 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n"] Oct 29 23:30:00.294402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.292852 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb"] Oct 29 23:30:00.294402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.292877 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh"] Oct 29 23:30:00.294402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.293772 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:00.302387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.302362 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx"] Oct 29 23:30:00.308723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.308456 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.311910 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhr5w\" (UniqueName: \"kubernetes.io/projected/8d03a3d3-acce-41e7-97ff-47c0beb3a320-kube-api-access-lhr5w\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv\" (UID: \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.311960 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk48p\" (UniqueName: \"kubernetes.io/projected/470d3dc9-0310-418e-be47-56549cb7573d-kube-api-access-sk48p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7\" (UID: \"470d3dc9-0310-418e-be47-56549cb7573d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.312021 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/470d3dc9-0310-418e-be47-56549cb7573d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7\" (UID: \"470d3dc9-0310-418e-be47-56549cb7573d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.312055 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njftq\" (UniqueName: \"kubernetes.io/projected/992c6925-45f5-4cb3-a436-f032da9b30f5-kube-api-access-njftq\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2\" (UID: \"992c6925-45f5-4cb3-a436-f032da9b30f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.312086 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/95c61621-cfc4-48fb-ad57-b77f44df64fa-kube-api-access-57gkt\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7\" (UID: \"95c61621-cfc4-48fb-ad57-b77f44df64fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.312181 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d3c108-ba23-4291-a0cf-909c58d2a8bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj\" (UID: \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.312697 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwtw\" (UniqueName: \"kubernetes.io/projected/b8868c3b-f832-4b4c-8968-139096f19a4a-kube-api-access-8vwtw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n\" (UID: \"b8868c3b-f832-4b4c-8968-139096f19a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.313011 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d3c108-ba23-4291-a0cf-909c58d2a8bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj\" (UID: \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.313248 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19f4669-9c01-4e64-a396-b755fdbfddd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2\" (UID: \"c19f4669-9c01-4e64-a396-b755fdbfddd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.313739 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c6925-45f5-4cb3-a436-f032da9b30f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2\" (UID: \"992c6925-45f5-4cb3-a436-f032da9b30f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.313803 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnws7\" (UniqueName: \"kubernetes.io/projected/2ab2f333-8e37-4019-ae18-e4a748837c22-kube-api-access-nnws7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb\" (UID: \"2ab2f333-8e37-4019-ae18-e4a748837c22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.313862 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76f9f99-0971-49bb-b819-23ce4f897001-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb\" (UID: \"f76f9f99-0971-49bb-b819-23ce4f897001\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.313875 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19f4669-9c01-4e64-a396-b755fdbfddd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2\" (UID: \"c19f4669-9c01-4e64-a396-b755fdbfddd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.313986 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8868c3b-f832-4b4c-8968-139096f19a4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n\" (UID: \"b8868c3b-f832-4b4c-8968-139096f19a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.314017 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjdz\" (UniqueName: \"kubernetes.io/projected/08d3c108-ba23-4291-a0cf-909c58d2a8bb-kube-api-access-tmjdz\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj\" (UID: \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.314051 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p42v\" (UniqueName: \"kubernetes.io/projected/c19f4669-9c01-4e64-a396-b755fdbfddd6-kube-api-access-4p42v\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2\" (UID: \"c19f4669-9c01-4e64-a396-b755fdbfddd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.314095 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03a3d3-acce-41e7-97ff-47c0beb3a320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv\" (UID: \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.314122 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f76f9f99-0971-49bb-b819-23ce4f897001-kube-api-access-2x4lx\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb\" (UID: \"f76f9f99-0971-49bb-b819-23ce4f897001\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.314172 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab2f333-8e37-4019-ae18-e4a748837c22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb\" (UID: \"2ab2f333-8e37-4019-ae18-e4a748837c22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.314210 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c61621-cfc4-48fb-ad57-b77f44df64fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7\" (UID: \"95c61621-cfc4-48fb-ad57-b77f44df64fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.314542 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76f9f99-0971-49bb-b819-23ce4f897001-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb\" (UID: \"f76f9f99-0971-49bb-b819-23ce4f897001\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.315163 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8868c3b-f832-4b4c-8968-139096f19a4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n\" (UID: \"b8868c3b-f832-4b4c-8968-139096f19a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.315307 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c6925-45f5-4cb3-a436-f032da9b30f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2\" (UID: \"992c6925-45f5-4cb3-a436-f032da9b30f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:00.317403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.315404 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03a3d3-acce-41e7-97ff-47c0beb3a320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv\" (UID: \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:00.320157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.318279 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx"] Oct 29 23:30:00.320157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.318379 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:00.325288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.325266 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d"] Oct 29 23:30:00.345061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.345035 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx"] Oct 29 23:30:00.345259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.345077 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh"] Oct 29 23:30:00.345259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.345103 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d"] Oct 29 23:30:00.345259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.345195 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:00.350959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.350939 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw"] Oct 29 23:30:00.382422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.382378 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb"] Oct 29 23:30:00.383088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.382496 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:00.393937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.393915 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5"] Oct 29 23:30:00.396495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.396475 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:00.420645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.415928 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjdz\" (UniqueName: \"kubernetes.io/projected/08d3c108-ba23-4291-a0cf-909c58d2a8bb-kube-api-access-tmjdz\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj\" (UID: \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:00.431637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.423338 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhr5w\" (UniqueName: \"kubernetes.io/projected/8d03a3d3-acce-41e7-97ff-47c0beb3a320-kube-api-access-lhr5w\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv\" (UID: \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:00.436629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.426469 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk48p\" (UniqueName: \"kubernetes.io/projected/470d3dc9-0310-418e-be47-56549cb7573d-kube-api-access-sk48p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7\" (UID: \"470d3dc9-0310-418e-be47-56549cb7573d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:00.436629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.434630 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/470d3dc9-0310-418e-be47-56549cb7573d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7\" (UID: \"470d3dc9-0310-418e-be47-56549cb7573d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:00.436629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.434777 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/95c61621-cfc4-48fb-ad57-b77f44df64fa-kube-api-access-57gkt\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7\" (UID: \"95c61621-cfc4-48fb-ad57-b77f44df64fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:00.436629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.434847 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw"] Oct 29 23:30:00.436629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.434902 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd"] Oct 29 23:30:00.436629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.435705 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz92v\" (UniqueName: \"kubernetes.io/projected/a7da5bb7-45be-4fc5-94d9-de6402481acb-kube-api-access-pz92v\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx\" (UID: \"a7da5bb7-45be-4fc5-94d9-de6402481acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:00.439553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.436778 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25smj\" (UniqueName: \"kubernetes.io/projected/48af7825-6733-4565-8347-58c931a08499-kube-api-access-25smj\") pod \"maas-api-key-cleanup-29888610-qfk7d\" (UID: \"48af7825-6733-4565-8347-58c931a08499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:00.439553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.437498 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srr5r\" (UniqueName: \"kubernetes.io/projected/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-kube-api-access-srr5r\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw\" (UID: \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:00.439553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.437774 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e77280-ca64-49f3-a6ca-ee1ec7573142-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx\" (UID: \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:00.439553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.438059 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48af7825-6733-4565-8347-58c931a08499-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-qfk7d\" (UID: \"48af7825-6733-4565-8347-58c931a08499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:00.439553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.438233 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw\" (UID: \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:00.439553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.438536 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnws7\" (UniqueName: \"kubernetes.io/projected/2ab2f333-8e37-4019-ae18-e4a748837c22-kube-api-access-nnws7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb\" (UID: \"2ab2f333-8e37-4019-ae18-e4a748837c22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:00.439553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.438694 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g6fw\" (UniqueName: \"kubernetes.io/projected/e1e77280-ca64-49f3-a6ca-ee1ec7573142-kube-api-access-4g6fw\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx\" (UID: \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:00.439553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.439047 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6hh\" (UniqueName: \"kubernetes.io/projected/8aad335b-fbf2-41ff-b597-ce071f6837e6-kube-api-access-hk6hh\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh\" (UID: \"8aad335b-fbf2-41ff-b597-ce071f6837e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:00.439553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.439361 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aad335b-fbf2-41ff-b597-ce071f6837e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh\" (UID: \"8aad335b-fbf2-41ff-b597-ce071f6837e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:00.441085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.440966 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7da5bb7-45be-4fc5-94d9-de6402481acb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx\" (UID: \"a7da5bb7-45be-4fc5-94d9-de6402481acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:00.441347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.441330 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:00.441427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.441394 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab2f333-8e37-4019-ae18-e4a748837c22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb\" (UID: \"2ab2f333-8e37-4019-ae18-e4a748837c22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:00.441704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.441681 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c61621-cfc4-48fb-ad57-b77f44df64fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7\" (UID: \"95c61621-cfc4-48fb-ad57-b77f44df64fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:00.444484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.444267 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p42v\" (UniqueName: \"kubernetes.io/projected/c19f4669-9c01-4e64-a396-b755fdbfddd6-kube-api-access-4p42v\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2\" (UID: \"c19f4669-9c01-4e64-a396-b755fdbfddd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:00.444654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.444627 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f76f9f99-0971-49bb-b819-23ce4f897001-kube-api-access-2x4lx\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb\" (UID: \"f76f9f99-0971-49bb-b819-23ce4f897001\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:00.445305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.429251 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwtw\" (UniqueName: \"kubernetes.io/projected/b8868c3b-f832-4b4c-8968-139096f19a4a-kube-api-access-8vwtw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n\" (UID: \"b8868c3b-f832-4b4c-8968-139096f19a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:00.448265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.428444 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njftq\" (UniqueName: \"kubernetes.io/projected/992c6925-45f5-4cb3-a436-f032da9b30f5-kube-api-access-njftq\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2\" (UID: \"992c6925-45f5-4cb3-a436-f032da9b30f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:00.451600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.451537 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab2f333-8e37-4019-ae18-e4a748837c22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb\" (UID: \"2ab2f333-8e37-4019-ae18-e4a748837c22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:00.451778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.451755 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/470d3dc9-0310-418e-be47-56549cb7573d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7\" (UID: \"470d3dc9-0310-418e-be47-56549cb7573d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:00.457824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.457799 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c61621-cfc4-48fb-ad57-b77f44df64fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7\" (UID: \"95c61621-cfc4-48fb-ad57-b77f44df64fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:00.464361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.464334 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk48p\" (UniqueName: \"kubernetes.io/projected/470d3dc9-0310-418e-be47-56549cb7573d-kube-api-access-sk48p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7\" (UID: \"470d3dc9-0310-418e-be47-56549cb7573d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:00.467649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.467627 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:00.473816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.473764 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb"] Oct 29 23:30:00.474533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.474268 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5"] Oct 29 23:30:00.474533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.474333 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp"] Oct 29 23:30:00.474533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.474389 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:00.480681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.480522 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd"] Oct 29 23:30:00.480775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.480727 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp"] Oct 29 23:30:00.480856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.480658 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:00.481478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.481449 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnws7\" (UniqueName: \"kubernetes.io/projected/2ab2f333-8e37-4019-ae18-e4a748837c22-kube-api-access-nnws7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb\" (UID: \"2ab2f333-8e37-4019-ae18-e4a748837c22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:00.483140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.483116 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/95c61621-cfc4-48fb-ad57-b77f44df64fa-kube-api-access-57gkt\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7\" (UID: \"95c61621-cfc4-48fb-ad57-b77f44df64fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:00.517460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.517436 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.546319 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.546841 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb\" (UID: \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.546926 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz92v\" (UniqueName: \"kubernetes.io/projected/a7da5bb7-45be-4fc5-94d9-de6402481acb-kube-api-access-pz92v\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx\" (UID: \"a7da5bb7-45be-4fc5-94d9-de6402481acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.546965 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25smj\" (UniqueName: \"kubernetes.io/projected/48af7825-6733-4565-8347-58c931a08499-kube-api-access-25smj\") pod \"maas-api-key-cleanup-29888610-qfk7d\" (UID: \"48af7825-6733-4565-8347-58c931a08499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547002 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r22q\" (UniqueName: \"kubernetes.io/projected/ffbe9920-f212-4ec5-8055-754d35f7e76e-kube-api-access-7r22q\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5\" (UID: \"ffbe9920-f212-4ec5-8055-754d35f7e76e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547034 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srr5r\" (UniqueName: \"kubernetes.io/projected/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-kube-api-access-srr5r\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw\" (UID: \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547064 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e77280-ca64-49f3-a6ca-ee1ec7573142-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx\" (UID: \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547143 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48af7825-6733-4565-8347-58c931a08499-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-qfk7d\" (UID: \"48af7825-6733-4565-8347-58c931a08499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547178 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw\" (UID: \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547282 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g6fw\" (UniqueName: \"kubernetes.io/projected/e1e77280-ca64-49f3-a6ca-ee1ec7573142-kube-api-access-4g6fw\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx\" (UID: \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547373 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6hh\" (UniqueName: \"kubernetes.io/projected/8aad335b-fbf2-41ff-b597-ce071f6837e6-kube-api-access-hk6hh\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh\" (UID: \"8aad335b-fbf2-41ff-b597-ce071f6837e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547408 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aad335b-fbf2-41ff-b597-ce071f6837e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh\" (UID: \"8aad335b-fbf2-41ff-b597-ce071f6837e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547450 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cab44-e6f4-44b3-be68-2e14816aeb27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd\" (UID: \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547525 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7da5bb7-45be-4fc5-94d9-de6402481acb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx\" (UID: \"a7da5bb7-45be-4fc5-94d9-de6402481acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547614 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp\" (UID: \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547648 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbe9920-f212-4ec5-8055-754d35f7e76e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5\" (UID: \"ffbe9920-f212-4ec5-8055-754d35f7e76e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547683 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42fp\" (UniqueName: \"kubernetes.io/projected/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-kube-api-access-k42fp\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb\" (UID: \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547704 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlljf\" (UniqueName: \"kubernetes.io/projected/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-kube-api-access-rlljf\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp\" (UID: \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:00.548042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.547726 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drtwc\" (UniqueName: \"kubernetes.io/projected/ca0cab44-e6f4-44b3-be68-2e14816aeb27-kube-api-access-drtwc\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd\" (UID: \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:00.549240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.548923 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aad335b-fbf2-41ff-b597-ce071f6837e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh\" (UID: \"8aad335b-fbf2-41ff-b597-ce071f6837e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:00.549240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.549189 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48af7825-6733-4565-8347-58c931a08499-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-qfk7d\" (UID: \"48af7825-6733-4565-8347-58c931a08499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:00.549512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.549372 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7da5bb7-45be-4fc5-94d9-de6402481acb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx\" (UID: \"a7da5bb7-45be-4fc5-94d9-de6402481acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:00.550696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.550026 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw\" (UID: \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:00.550696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.550444 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:00.550696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.550508 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e77280-ca64-49f3-a6ca-ee1ec7573142-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx\" (UID: \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:00.567946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.560507 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:00.581554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.581286 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:00.592942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.592870 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g6fw\" (UniqueName: \"kubernetes.io/projected/e1e77280-ca64-49f3-a6ca-ee1ec7573142-kube-api-access-4g6fw\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx\" (UID: \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:00.593949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.593909 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srr5r\" (UniqueName: \"kubernetes.io/projected/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-kube-api-access-srr5r\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw\" (UID: \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:00.594521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.594484 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6hh\" (UniqueName: \"kubernetes.io/projected/8aad335b-fbf2-41ff-b597-ce071f6837e6-kube-api-access-hk6hh\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh\" (UID: \"8aad335b-fbf2-41ff-b597-ce071f6837e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:00.595306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.595277 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25smj\" (UniqueName: \"kubernetes.io/projected/48af7825-6733-4565-8347-58c931a08499-kube-api-access-25smj\") pod \"maas-api-key-cleanup-29888610-qfk7d\" (UID: \"48af7825-6733-4565-8347-58c931a08499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:00.597192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.597139 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz92v\" (UniqueName: \"kubernetes.io/projected/a7da5bb7-45be-4fc5-94d9-de6402481acb-kube-api-access-pz92v\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx\" (UID: \"a7da5bb7-45be-4fc5-94d9-de6402481acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:00.618017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.617996 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:00.621406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.621224 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:00.650475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.650433 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp\" (UID: \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:00.650653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.650492 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbe9920-f212-4ec5-8055-754d35f7e76e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5\" (UID: \"ffbe9920-f212-4ec5-8055-754d35f7e76e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:00.650653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.650536 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42fp\" (UniqueName: \"kubernetes.io/projected/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-kube-api-access-k42fp\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb\" (UID: \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:00.650653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.650569 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlljf\" (UniqueName: \"kubernetes.io/projected/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-kube-api-access-rlljf\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp\" (UID: \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:00.650836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.650657 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drtwc\" (UniqueName: \"kubernetes.io/projected/ca0cab44-e6f4-44b3-be68-2e14816aeb27-kube-api-access-drtwc\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd\" (UID: \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:00.650894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.650838 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb\" (UID: \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:00.650953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.650926 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r22q\" (UniqueName: \"kubernetes.io/projected/ffbe9920-f212-4ec5-8055-754d35f7e76e-kube-api-access-7r22q\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5\" (UID: \"ffbe9920-f212-4ec5-8055-754d35f7e76e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:00.676619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.651260 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cab44-e6f4-44b3-be68-2e14816aeb27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd\" (UID: \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:00.676619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.653334 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbe9920-f212-4ec5-8055-754d35f7e76e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5\" (UID: \"ffbe9920-f212-4ec5-8055-754d35f7e76e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:00.676619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.654600 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb\" (UID: \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:00.676619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.661978 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cab44-e6f4-44b3-be68-2e14816aeb27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd\" (UID: \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:00.676619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.663237 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp\" (UID: \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:00.741384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.733790 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42fp\" (UniqueName: \"kubernetes.io/projected/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-kube-api-access-k42fp\") pod \"maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb\" (UID: \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:00.741384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.734834 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlljf\" (UniqueName: \"kubernetes.io/projected/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-kube-api-access-rlljf\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp\" (UID: \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:00.741384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.734928 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drtwc\" (UniqueName: \"kubernetes.io/projected/ca0cab44-e6f4-44b3-be68-2e14816aeb27-kube-api-access-drtwc\") pod \"maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd\" (UID: \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:00.741384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.737442 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r22q\" (UniqueName: \"kubernetes.io/projected/ffbe9920-f212-4ec5-8055-754d35f7e76e-kube-api-access-7r22q\") pod \"maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5\" (UID: \"ffbe9920-f212-4ec5-8055-754d35f7e76e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:00.772129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.771271 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:00.796522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.796303 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:00.810875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.807804 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:00.819623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.818984 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:00.834154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.834121 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:00.842466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.842166 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:00.885601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.880805 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:00.885601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.881296 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:00.911133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.910963 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:00.918931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:00.918902 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:01.084118 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:01.056990 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d03a3d3_acce_41e7_97ff_47c0beb3a320.slice/crio-6af5687abfc3d2643cb149e68d2f2afb112b280048bb538f349490df8bbb8230 WatchSource:0}: Error finding container 6af5687abfc3d2643cb149e68d2f2afb112b280048bb538f349490df8bbb8230: Status 404 returned error can't find the container with id 6af5687abfc3d2643cb149e68d2f2afb112b280048bb538f349490df8bbb8230 Oct 29 23:30:01.084118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.078633 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" podUID="baa1e792-c7be-446e-850e-a645eb1f9df9" containerName="payload-processing" containerID="cri-o://69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a" gracePeriod=25 Oct 29 23:30:01.133093 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.133029 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd129ecb-6995-4dfc-bda1-4da0b20a5a79" path="/var/lib/kubelet/pods/cd129ecb-6995-4dfc-bda1-4da0b20a5a79/volumes" Oct 29 23:30:01.133736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.133656 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv"] Oct 29 23:30:01.259665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.259627 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj"] Oct 29 23:30:01.335175 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:01.335126 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d3c108_ba23_4291_a0cf_909c58d2a8bb.slice/crio-e140b35e41b46f6e6b55a68bbcc82530e9b1794751a411aaf1f3d2aeca25cb69 WatchSource:0}: Error finding container e140b35e41b46f6e6b55a68bbcc82530e9b1794751a411aaf1f3d2aeca25cb69: Status 404 returned error can't find the container with id e140b35e41b46f6e6b55a68bbcc82530e9b1794751a411aaf1f3d2aeca25cb69 Oct 29 23:30:01.446030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.445992 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb"] Oct 29 23:30:01.465537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.465500 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2"] Oct 29 23:30:01.465925 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:01.465898 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76f9f99_0971_49bb_b819_23ce4f897001.slice/crio-78707e26728ace64c418aa7bf2e2dbb0deebf40b6dd6f5ffbba29f0cdcd2cf5b WatchSource:0}: Error finding container 78707e26728ace64c418aa7bf2e2dbb0deebf40b6dd6f5ffbba29f0cdcd2cf5b: Status 404 returned error can't find the container with id 78707e26728ace64c418aa7bf2e2dbb0deebf40b6dd6f5ffbba29f0cdcd2cf5b Oct 29 23:30:01.470484 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:01.470400 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992c6925_45f5_4cb3_a436_f032da9b30f5.slice/crio-1b87f73df0e82d766400a31007af9b15aab52adb7b7583cfded25979ec14cb8b WatchSource:0}: Error finding container 1b87f73df0e82d766400a31007af9b15aab52adb7b7583cfded25979ec14cb8b: Status 404 returned error can't find the container with id 1b87f73df0e82d766400a31007af9b15aab52adb7b7583cfded25979ec14cb8b Oct 29 23:30:01.499255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.499225 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7"] Oct 29 23:30:01.524733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.516709 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n"] Oct 29 23:30:01.611054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.611028 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:30:01.615611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.615555 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp"] Oct 29 23:30:01.618710 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:01.618684 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b998c5e_6c2a_4065_898e_e4ba61b45a0a.slice/crio-2e33597b6db65a88943dfc7f9de502867f5974db66b01ce807cd17cd3c8ae52d WatchSource:0}: Error finding container 2e33597b6db65a88943dfc7f9de502867f5974db66b01ce807cd17cd3c8ae52d: Status 404 returned error can't find the container with id 2e33597b6db65a88943dfc7f9de502867f5974db66b01ce807cd17cd3c8ae52d Oct 29 23:30:01.631361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.631333 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2"] Oct 29 23:30:01.640774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.640745 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5"] Oct 29 23:30:01.651830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.651807 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7"] Oct 29 23:30:01.657029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.656958 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh"] Oct 29 23:30:01.699643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.699607 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx"] Oct 29 23:30:01.720322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.720285 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-routing-overlay-volume\") pod \"baa1e792-c7be-446e-850e-a645eb1f9df9\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " Oct 29 23:30:01.720527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.720481 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-provider-credentials\") pod \"baa1e792-c7be-446e-850e-a645eb1f9df9\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " Oct 29 23:30:01.720623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.720531 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-kube-api-access-slhtm\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-kube-api-access-slhtm\") pod \"baa1e792-c7be-446e-850e-a645eb1f9df9\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " Oct 29 23:30:01.720684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.720620 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-routing-overlay-volume" pod "baa1e792-c7be-446e-850e-a645eb1f9df9" (UID: "baa1e792-c7be-446e-850e-a645eb1f9df9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:01.720684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.720633 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-plugins-config-volume\") pod \"baa1e792-c7be-446e-850e-a645eb1f9df9\" (UID: \"baa1e792-c7be-446e-850e-a645eb1f9df9\") " Oct 29 23:30:01.721490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.721459 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-plugins-config-volume" pod "baa1e792-c7be-446e-850e-a645eb1f9df9" (UID: "baa1e792-c7be-446e-850e-a645eb1f9df9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:01.721632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.721557 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:01.737616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.737550 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-kube-api-access-slhtm" pod "baa1e792-c7be-446e-850e-a645eb1f9df9" (UID: "baa1e792-c7be-446e-850e-a645eb1f9df9"). InnerVolumeSpecName "kube-api-access-slhtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:01.737739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.737686 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-provider-credentials" pod "baa1e792-c7be-446e-850e-a645eb1f9df9" (UID: "baa1e792-c7be-446e-850e-a645eb1f9df9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:01.753831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.753806 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd"] Oct 29 23:30:01.766480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.766452 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb"] Oct 29 23:30:01.777614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.777572 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw"] Oct 29 23:30:01.787023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.786993 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx"] Oct 29 23:30:01.791153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.791130 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d"] Oct 29 23:30:01.807395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.807372 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb"] Oct 29 23:30:01.823650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.823576 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:01.823650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.823651 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slhtm\" (UniqueName: \"kubernetes.io/projected/baa1e792-c7be-446e-850e-a645eb1f9df9-kube-api-access-slhtm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:01.823836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.823661 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa1e792-c7be-446e-850e-a645eb1f9df9-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:01.841481 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:01.841436 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aad335b_fbf2_41ff_b597_ce071f6837e6.slice/crio-ee1e2f0b62497841b3a213a5f321ff0c9ca1b3fd1297090ba362cd3d46611cd2 WatchSource:0}: Error finding container ee1e2f0b62497841b3a213a5f321ff0c9ca1b3fd1297090ba362cd3d46611cd2: Status 404 returned error can't find the container with id ee1e2f0b62497841b3a213a5f321ff0c9ca1b3fd1297090ba362cd3d46611cd2 Oct 29 23:30:01.841922 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:01.841896 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7da5bb7_45be_4fc5_94d9_de6402481acb.slice/crio-8fc502e154d347778f44fff27e3eaa800168c3f0a57897228e1314a2fb494304 WatchSource:0}: Error finding container 8fc502e154d347778f44fff27e3eaa800168c3f0a57897228e1314a2fb494304: Status 404 returned error can't find the container with id 8fc502e154d347778f44fff27e3eaa800168c3f0a57897228e1314a2fb494304 Oct 29 23:30:01.847313 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:01.843336 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46519c4c_5baa_453c_8e06_d2c1e41eb6dd.slice/crio-5f7dc4aab5e1ebad95b298f892b8e83397cc3ebbdeddb8e9e9d3e6d1830d01d2 WatchSource:0}: Error finding container 5f7dc4aab5e1ebad95b298f892b8e83397cc3ebbdeddb8e9e9d3e6d1830d01d2: Status 404 returned error can't find the container with id 5f7dc4aab5e1ebad95b298f892b8e83397cc3ebbdeddb8e9e9d3e6d1830d01d2 Oct 29 23:30:01.973217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.973136 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" event={"ID":"48af7825-6733-4565-8347-58c931a08499","Type":"ContainerStarted","Data":"fd5c6aa02ac6edfba46b37f3747ed7d72fad788928d858eb686718703d8c39bf"} Oct 29 23:30:01.974538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.974509 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" event={"ID":"c19f4669-9c01-4e64-a396-b755fdbfddd6","Type":"ContainerStarted","Data":"ef0bd1e148e0825a9a25db6e185b6f146bf9a2bf94c0f253c192a51915286996"} Oct 29 23:30:01.975799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.975773 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" event={"ID":"46519c4c-5baa-453c-8e06-d2c1e41eb6dd","Type":"ContainerStarted","Data":"5f7dc4aab5e1ebad95b298f892b8e83397cc3ebbdeddb8e9e9d3e6d1830d01d2"} Oct 29 23:30:01.977048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.977024 5736 generic.go:356] "Generic (PLEG): container finished" podID="baa1e792-c7be-446e-850e-a645eb1f9df9" containerID="69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a" exitCode=0 Oct 29 23:30:01.977121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.977076 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" Oct 29 23:30:01.977121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.977106 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" event={"ID":"baa1e792-c7be-446e-850e-a645eb1f9df9","Type":"ContainerDied","Data":"69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a"} Oct 29 23:30:01.977246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.977149 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst" event={"ID":"baa1e792-c7be-446e-850e-a645eb1f9df9","Type":"ContainerDied","Data":"8956e6916117e607741d4e2aa24caf11422cd27a8f417798667dfc543749bcb8"} Oct 29 23:30:01.977246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.977162 5736 scope.go:122] "RemoveContainer" containerID="69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a" Oct 29 23:30:01.978938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.978911 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" event={"ID":"470d3dc9-0310-418e-be47-56549cb7573d","Type":"ContainerStarted","Data":"13d2e6ecf500a21ba6eea8db2b3dcc88b0e9afe4abc52a33bed056bde176a490"} Oct 29 23:30:01.980377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.980353 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" event={"ID":"ca0cab44-e6f4-44b3-be68-2e14816aeb27","Type":"ContainerStarted","Data":"3aec4d584cd76d0ce3271822c3b12a0156dac334c4992d0ec879f9ab43f4a5a5"} Oct 29 23:30:01.981759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.981728 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" event={"ID":"95c61621-cfc4-48fb-ad57-b77f44df64fa","Type":"ContainerStarted","Data":"3f78472645647b22c64774d9c97c828a41afe75de7382db0eee760764856a240"} Oct 29 23:30:01.983201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.983178 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" event={"ID":"a7da5bb7-45be-4fc5-94d9-de6402481acb","Type":"ContainerStarted","Data":"8fc502e154d347778f44fff27e3eaa800168c3f0a57897228e1314a2fb494304"} Oct 29 23:30:01.984405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.984384 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" event={"ID":"e1e77280-ca64-49f3-a6ca-ee1ec7573142","Type":"ContainerStarted","Data":"5b4b015d0b0fc2d087e4726c8a7516923d83b3a53ee826213651218d1b6dbb36"} Oct 29 23:30:01.985786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.985716 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" event={"ID":"f76f9f99-0971-49bb-b819-23ce4f897001","Type":"ContainerStarted","Data":"78707e26728ace64c418aa7bf2e2dbb0deebf40b6dd6f5ffbba29f0cdcd2cf5b"} Oct 29 23:30:01.987101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.987072 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" event={"ID":"ffbe9920-f212-4ec5-8055-754d35f7e76e","Type":"ContainerStarted","Data":"4d1e6c5fd1d88f794bc1b80e09d0ec42181cc33bd2454ed4424dfd325c3a409d"} Oct 29 23:30:01.988302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.988279 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" event={"ID":"08d3c108-ba23-4291-a0cf-909c58d2a8bb","Type":"ContainerStarted","Data":"e140b35e41b46f6e6b55a68bbcc82530e9b1794751a411aaf1f3d2aeca25cb69"} Oct 29 23:30:01.989427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.989407 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" event={"ID":"8aad335b-fbf2-41ff-b597-ce071f6837e6","Type":"ContainerStarted","Data":"ee1e2f0b62497841b3a213a5f321ff0c9ca1b3fd1297090ba362cd3d46611cd2"} Oct 29 23:30:01.991085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.991059 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" event={"ID":"8d03a3d3-acce-41e7-97ff-47c0beb3a320","Type":"ContainerStarted","Data":"6075f26a474032745bef9ad2eaf14a10052dabebb4a9d7a92f390ce8188901ab"} Oct 29 23:30:01.991375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.991101 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" event={"ID":"8d03a3d3-acce-41e7-97ff-47c0beb3a320","Type":"ContainerStarted","Data":"6af5687abfc3d2643cb149e68d2f2afb112b280048bb538f349490df8bbb8230"} Oct 29 23:30:01.992309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.992287 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" event={"ID":"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2","Type":"ContainerStarted","Data":"93fd97653e14a04d6a88b9f9b9674b587180fcf32cc5a973443d7f56da87aa22"} Oct 29 23:30:01.993466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.993445 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" event={"ID":"b8868c3b-f832-4b4c-8968-139096f19a4a","Type":"ContainerStarted","Data":"215ee9656340eb3f37d72b7be8a6a37ba97ff0e910d935f70ed8eb83c99defa3"} Oct 29 23:30:01.994635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.994573 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" event={"ID":"2b998c5e-6c2a-4065-898e-e4ba61b45a0a","Type":"ContainerStarted","Data":"2e33597b6db65a88943dfc7f9de502867f5974db66b01ce807cd17cd3c8ae52d"} Oct 29 23:30:01.995707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.995686 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" event={"ID":"2ab2f333-8e37-4019-ae18-e4a748837c22","Type":"ContainerStarted","Data":"af8fbeeef9da56e7851a64d2e9ea4c7b8e02c36c8b4d826d0f0422fa01c158d0"} Oct 29 23:30:01.997711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:01.997688 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" event={"ID":"992c6925-45f5-4cb3-a436-f032da9b30f5","Type":"ContainerStarted","Data":"1b87f73df0e82d766400a31007af9b15aab52adb7b7583cfded25979ec14cb8b"} Oct 29 23:30:02.035433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:02.035362 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst"] Oct 29 23:30:02.040495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:02.040470 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-75767nzst"] Oct 29 23:30:02.063539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:02.063496 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t"] Oct 29 23:30:02.087153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:02.081949 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" podStartSLOduration=2.081929764 podStartE2EDuration="2.081929764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:02.07984812 +0000 UTC m=+2799.195767723" watchObservedRunningTime="2026-10-29 23:30:02.081929764 +0000 UTC m=+2799.197849372" Oct 29 23:30:02.087153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:02.082770 5736 scope.go:122] "RemoveContainer" containerID="69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a" Oct 29 23:30:02.087153 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:02.084185 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a\": container with ID starting with 69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a not found: ID does not exist" containerID="69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a" Oct 29 23:30:02.087153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:02.084225 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a"} err="failed to get container status \"69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a\": rpc error: code = NotFound desc = could not find container \"69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a\": container with ID starting with 69fdc974a8a74042a13681f3c920f08751d09482ff0ab836f66e3dc1136b0f2a not found: ID does not exist" Oct 29 23:30:03.027265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.026839 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa1e792-c7be-446e-850e-a645eb1f9df9" path="/var/lib/kubelet/pods/baa1e792-c7be-446e-850e-a645eb1f9df9/volumes" Oct 29 23:30:03.050241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.050194 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" event={"ID":"48af7825-6733-4565-8347-58c931a08499","Type":"ContainerStarted","Data":"446a7f5f406c6933d4db22fbb97ab013dce99ba9f643c12affdc3fc7c9b35f90"} Oct 29 23:30:03.054026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.053971 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" event={"ID":"c19f4669-9c01-4e64-a396-b755fdbfddd6","Type":"ContainerStarted","Data":"60eeaa47e85eb02c6d05ca3d91ef00fce7c0617057bee69e59dbd6b2dab6e237"} Oct 29 23:30:03.060625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.060573 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" event={"ID":"470d3dc9-0310-418e-be47-56549cb7573d","Type":"ContainerStarted","Data":"945b8924ecbde68eaf4c78567bf2362689615d6e8766735033f63fa5084feb29"} Oct 29 23:30:03.063640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.063436 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" event={"ID":"95c61621-cfc4-48fb-ad57-b77f44df64fa","Type":"ContainerStarted","Data":"3096cc3b98256ba2c521a1e897218e89906e9bd536cf98ab0509fdf5bcce77c7"} Oct 29 23:30:03.067041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.066515 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" event={"ID":"f76f9f99-0971-49bb-b819-23ce4f897001","Type":"ContainerStarted","Data":"ea4edd537dda8fc9beb03ca34ce180479681712b55ea7f19d1808c4ba0267347"} Oct 29 23:30:03.069284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.069255 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" event={"ID":"ffbe9920-f212-4ec5-8055-754d35f7e76e","Type":"ContainerStarted","Data":"f6f078496544220063d49f43246d715160c20b6845f07b77c8dfc7460a4c028c"} Oct 29 23:30:03.071082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.071053 5736 generic.go:356] "Generic (PLEG): container finished" podID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerID="cff090a11be52bcd7e112ac8e79a8d0861cd2389095c62903f78b661c562dd1e" exitCode=60 Oct 29 23:30:03.071195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.071097 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" event={"ID":"08d3c108-ba23-4291-a0cf-909c58d2a8bb","Type":"ContainerDied","Data":"cff090a11be52bcd7e112ac8e79a8d0861cd2389095c62903f78b661c562dd1e"} Oct 29 23:30:03.071368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.071351 5736 scope.go:122] "RemoveContainer" containerID="cff090a11be52bcd7e112ac8e79a8d0861cd2389095c62903f78b661c562dd1e" Oct 29 23:30:03.073474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.073451 5736 generic.go:356] "Generic (PLEG): container finished" podID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerID="6075f26a474032745bef9ad2eaf14a10052dabebb4a9d7a92f390ce8188901ab" exitCode=60 Oct 29 23:30:03.073664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.073523 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" event={"ID":"8d03a3d3-acce-41e7-97ff-47c0beb3a320","Type":"ContainerDied","Data":"6075f26a474032745bef9ad2eaf14a10052dabebb4a9d7a92f390ce8188901ab"} Oct 29 23:30:03.073724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.073695 5736 scope.go:122] "RemoveContainer" containerID="6075f26a474032745bef9ad2eaf14a10052dabebb4a9d7a92f390ce8188901ab" Oct 29 23:30:03.075436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.075412 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" event={"ID":"b8868c3b-f832-4b4c-8968-139096f19a4a","Type":"ContainerStarted","Data":"992eedcaa5178a6eabcae206d9a76d765cae656c7a080cf0692d70266793854f"} Oct 29 23:30:03.077284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.077261 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" event={"ID":"2b998c5e-6c2a-4065-898e-e4ba61b45a0a","Type":"ContainerStarted","Data":"abca25941bfc5b7e55c83d6c0a758647af8401fc3ed96d3bf798e01eca26ed61"} Oct 29 23:30:03.079289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.079266 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" event={"ID":"2ab2f333-8e37-4019-ae18-e4a748837c22","Type":"ContainerStarted","Data":"b4eb02c3ca32447447f6987024e1f7642220920360b98f3260309faf7740fa6a"} Oct 29 23:30:03.081617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.081571 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" event={"ID":"992c6925-45f5-4cb3-a436-f032da9b30f5","Type":"ContainerStarted","Data":"ce2a5b91c838b360ec9a2c6746ba1edad29635fe36811d0c3e7aa650927d383d"} Oct 29 23:30:03.091741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.091695 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" podStartSLOduration=3.091665237 podStartE2EDuration="3.091665237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.090052821 +0000 UTC m=+2800.205972428" watchObservedRunningTime="2026-10-29 23:30:03.091665237 +0000 UTC m=+2800.207584843" Oct 29 23:30:03.136252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.131760 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" podStartSLOduration=3.13174103 podStartE2EDuration="3.13174103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.126401215 +0000 UTC m=+2800.242320821" watchObservedRunningTime="2026-10-29 23:30:03.13174103 +0000 UTC m=+2800.247660638" Oct 29 23:30:03.167107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.167054 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" podStartSLOduration=3.167036834 podStartE2EDuration="3.167036834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.16251036 +0000 UTC m=+2800.278429966" watchObservedRunningTime="2026-10-29 23:30:03.167036834 +0000 UTC m=+2800.282956439" Oct 29 23:30:03.203766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.203698 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" podStartSLOduration=3.20342687 podStartE2EDuration="3.20342687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.202337624 +0000 UTC m=+2800.318257231" watchObservedRunningTime="2026-10-29 23:30:03.20342687 +0000 UTC m=+2800.319346485" Oct 29 23:30:03.276621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.276529 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" podStartSLOduration=3.276515057 podStartE2EDuration="3.276515057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.274664727 +0000 UTC m=+2800.390584336" watchObservedRunningTime="2026-10-29 23:30:03.276515057 +0000 UTC m=+2800.392434685" Oct 29 23:30:03.312856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.312785 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" podStartSLOduration=3.3127728850000002 podStartE2EDuration="3.312772885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.312137101 +0000 UTC m=+2800.428056704" watchObservedRunningTime="2026-10-29 23:30:03.312772885 +0000 UTC m=+2800.428692491" Oct 29 23:30:03.350353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.350294 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" podStartSLOduration=3.350282574 podStartE2EDuration="3.350282574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.349275791 +0000 UTC m=+2800.465195398" watchObservedRunningTime="2026-10-29 23:30:03.350282574 +0000 UTC m=+2800.466202179" Oct 29 23:30:03.405614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.403494 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" podStartSLOduration=3.403472527 podStartE2EDuration="3.403472527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.389301356 +0000 UTC m=+2800.505220963" watchObservedRunningTime="2026-10-29 23:30:03.403472527 +0000 UTC m=+2800.519392133" Oct 29 23:30:03.510203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.509267 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" podStartSLOduration=3.509253664 podStartE2EDuration="3.509253664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.507027733 +0000 UTC m=+2800.622947340" watchObservedRunningTime="2026-10-29 23:30:03.509253664 +0000 UTC m=+2800.625173269" Oct 29 23:30:03.554449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:03.551222 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" podStartSLOduration=3.55120303 podStartE2EDuration="3.55120303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.548899145 +0000 UTC m=+2800.664818754" watchObservedRunningTime="2026-10-29 23:30:03.55120303 +0000 UTC m=+2800.667122639" Oct 29 23:30:03.627197 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:03.627154 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b998c5e_6c2a_4065_898e_e4ba61b45a0a.slice/crio-abca25941bfc5b7e55c83d6c0a758647af8401fc3ed96d3bf798e01eca26ed61.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:30:04.223990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.223697 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" event={"ID":"e1e77280-ca64-49f3-a6ca-ee1ec7573142","Type":"ContainerStarted","Data":"8eab7d400b770b6d7d340f323f2ff2d18610bc23a27aeb8e15b86066e730d981"} Oct 29 23:30:04.231799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.225011 5736 scope.go:122] "RemoveContainer" containerID="8eab7d400b770b6d7d340f323f2ff2d18610bc23a27aeb8e15b86066e730d981" Oct 29 23:30:04.249405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.247322 5736 generic.go:356] "Generic (PLEG): container finished" podID="f76f9f99-0971-49bb-b819-23ce4f897001" containerID="ea4edd537dda8fc9beb03ca34ce180479681712b55ea7f19d1808c4ba0267347" exitCode=60 Oct 29 23:30:04.249405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.247442 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" event={"ID":"f76f9f99-0971-49bb-b819-23ce4f897001","Type":"ContainerDied","Data":"ea4edd537dda8fc9beb03ca34ce180479681712b55ea7f19d1808c4ba0267347"} Oct 29 23:30:04.249405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.247977 5736 scope.go:122] "RemoveContainer" containerID="ea4edd537dda8fc9beb03ca34ce180479681712b55ea7f19d1808c4ba0267347" Oct 29 23:30:04.254793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.252899 5736 generic.go:356] "Generic (PLEG): container finished" podID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerID="f6f078496544220063d49f43246d715160c20b6845f07b77c8dfc7460a4c028c" exitCode=60 Oct 29 23:30:04.254793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.253018 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" event={"ID":"ffbe9920-f212-4ec5-8055-754d35f7e76e","Type":"ContainerDied","Data":"f6f078496544220063d49f43246d715160c20b6845f07b77c8dfc7460a4c028c"} Oct 29 23:30:04.254793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.253518 5736 scope.go:122] "RemoveContainer" containerID="f6f078496544220063d49f43246d715160c20b6845f07b77c8dfc7460a4c028c" Oct 29 23:30:04.284469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.284440 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" event={"ID":"8aad335b-fbf2-41ff-b597-ce071f6837e6","Type":"ContainerStarted","Data":"84fce66d7083c25acc8ba0904455a8272a0b8709e92b06457566f47af1c63ebc"} Oct 29 23:30:04.284908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.284886 5736 scope.go:122] "RemoveContainer" containerID="84fce66d7083c25acc8ba0904455a8272a0b8709e92b06457566f47af1c63ebc" Oct 29 23:30:04.287369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.287349 5736 generic.go:356] "Generic (PLEG): container finished" podID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerID="fb1352e8cf33732c290240df9280e374d6df6c480260e5c103a2ead2fe4a5c95" exitCode=60 Oct 29 23:30:04.287554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.287536 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" event={"ID":"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2","Type":"ContainerDied","Data":"fb1352e8cf33732c290240df9280e374d6df6c480260e5c103a2ead2fe4a5c95"} Oct 29 23:30:04.287753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.287731 5736 scope.go:122] "RemoveContainer" containerID="fb1352e8cf33732c290240df9280e374d6df6c480260e5c103a2ead2fe4a5c95" Oct 29 23:30:04.291139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.291112 5736 generic.go:356] "Generic (PLEG): container finished" podID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerID="992eedcaa5178a6eabcae206d9a76d765cae656c7a080cf0692d70266793854f" exitCode=60 Oct 29 23:30:04.291244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.291225 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" event={"ID":"b8868c3b-f832-4b4c-8968-139096f19a4a","Type":"ContainerDied","Data":"992eedcaa5178a6eabcae206d9a76d765cae656c7a080cf0692d70266793854f"} Oct 29 23:30:04.291592 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.291562 5736 scope.go:122] "RemoveContainer" containerID="992eedcaa5178a6eabcae206d9a76d765cae656c7a080cf0692d70266793854f" Oct 29 23:30:04.293997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.293975 5736 generic.go:356] "Generic (PLEG): container finished" podID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerID="abca25941bfc5b7e55c83d6c0a758647af8401fc3ed96d3bf798e01eca26ed61" exitCode=60 Oct 29 23:30:04.294101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.294057 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" event={"ID":"2b998c5e-6c2a-4065-898e-e4ba61b45a0a","Type":"ContainerDied","Data":"abca25941bfc5b7e55c83d6c0a758647af8401fc3ed96d3bf798e01eca26ed61"} Oct 29 23:30:04.294376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.294359 5736 scope.go:122] "RemoveContainer" containerID="abca25941bfc5b7e55c83d6c0a758647af8401fc3ed96d3bf798e01eca26ed61" Oct 29 23:30:04.298134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.298106 5736 generic.go:356] "Generic (PLEG): container finished" podID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerID="b4eb02c3ca32447447f6987024e1f7642220920360b98f3260309faf7740fa6a" exitCode=60 Oct 29 23:30:04.298290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.298235 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" event={"ID":"2ab2f333-8e37-4019-ae18-e4a748837c22","Type":"ContainerDied","Data":"b4eb02c3ca32447447f6987024e1f7642220920360b98f3260309faf7740fa6a"} Oct 29 23:30:04.299998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.299963 5736 scope.go:122] "RemoveContainer" containerID="b4eb02c3ca32447447f6987024e1f7642220920360b98f3260309faf7740fa6a" Oct 29 23:30:04.304398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.304374 5736 generic.go:356] "Generic (PLEG): container finished" podID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerID="ce2a5b91c838b360ec9a2c6746ba1edad29635fe36811d0c3e7aa650927d383d" exitCode=60 Oct 29 23:30:04.304491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.304460 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" event={"ID":"992c6925-45f5-4cb3-a436-f032da9b30f5","Type":"ContainerDied","Data":"ce2a5b91c838b360ec9a2c6746ba1edad29635fe36811d0c3e7aa650927d383d"} Oct 29 23:30:04.304766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.304750 5736 scope.go:122] "RemoveContainer" containerID="ce2a5b91c838b360ec9a2c6746ba1edad29635fe36811d0c3e7aa650927d383d" Oct 29 23:30:04.349574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.349523 5736 generic.go:356] "Generic (PLEG): container finished" podID="48af7825-6733-4565-8347-58c931a08499" containerID="446a7f5f406c6933d4db22fbb97ab013dce99ba9f643c12affdc3fc7c9b35f90" exitCode=60 Oct 29 23:30:04.350732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.350704 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" event={"ID":"48af7825-6733-4565-8347-58c931a08499","Type":"ContainerDied","Data":"446a7f5f406c6933d4db22fbb97ab013dce99ba9f643c12affdc3fc7c9b35f90"} Oct 29 23:30:04.355531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.355503 5736 scope.go:122] "RemoveContainer" containerID="446a7f5f406c6933d4db22fbb97ab013dce99ba9f643c12affdc3fc7c9b35f90" Oct 29 23:30:04.360063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.359024 5736 generic.go:356] "Generic (PLEG): container finished" podID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerID="60eeaa47e85eb02c6d05ca3d91ef00fce7c0617057bee69e59dbd6b2dab6e237" exitCode=60 Oct 29 23:30:04.360063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.359127 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" event={"ID":"c19f4669-9c01-4e64-a396-b755fdbfddd6","Type":"ContainerDied","Data":"60eeaa47e85eb02c6d05ca3d91ef00fce7c0617057bee69e59dbd6b2dab6e237"} Oct 29 23:30:04.363196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.363172 5736 scope.go:122] "RemoveContainer" containerID="60eeaa47e85eb02c6d05ca3d91ef00fce7c0617057bee69e59dbd6b2dab6e237" Oct 29 23:30:04.372329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.372306 5736 generic.go:356] "Generic (PLEG): container finished" podID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerID="4169995f14e493b0d6e2844ad78e1b2ea9450c46b7dce4c306729a91eab9ce98" exitCode=60 Oct 29 23:30:04.372492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.372340 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" event={"ID":"46519c4c-5baa-453c-8e06-d2c1e41eb6dd","Type":"ContainerDied","Data":"4169995f14e493b0d6e2844ad78e1b2ea9450c46b7dce4c306729a91eab9ce98"} Oct 29 23:30:04.373931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.373908 5736 scope.go:122] "RemoveContainer" containerID="4169995f14e493b0d6e2844ad78e1b2ea9450c46b7dce4c306729a91eab9ce98" Oct 29 23:30:04.374854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.374833 5736 generic.go:356] "Generic (PLEG): container finished" podID="470d3dc9-0310-418e-be47-56549cb7573d" containerID="945b8924ecbde68eaf4c78567bf2362689615d6e8766735033f63fa5084feb29" exitCode=60 Oct 29 23:30:04.375203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.375188 5736 scope.go:122] "RemoveContainer" containerID="945b8924ecbde68eaf4c78567bf2362689615d6e8766735033f63fa5084feb29" Oct 29 23:30:04.375836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.375805 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" event={"ID":"470d3dc9-0310-418e-be47-56549cb7573d","Type":"ContainerDied","Data":"945b8924ecbde68eaf4c78567bf2362689615d6e8766735033f63fa5084feb29"} Oct 29 23:30:04.377253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.377230 5736 generic.go:356] "Generic (PLEG): container finished" podID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerID="1d9abccbf00aed42f600d28052f1ec5c7f4d91faac1fac60be5f0151289e9243" exitCode=60 Oct 29 23:30:04.377403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.377268 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" event={"ID":"ca0cab44-e6f4-44b3-be68-2e14816aeb27","Type":"ContainerDied","Data":"1d9abccbf00aed42f600d28052f1ec5c7f4d91faac1fac60be5f0151289e9243"} Oct 29 23:30:04.377651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.377629 5736 scope.go:122] "RemoveContainer" containerID="1d9abccbf00aed42f600d28052f1ec5c7f4d91faac1fac60be5f0151289e9243" Oct 29 23:30:04.388195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.387233 5736 generic.go:356] "Generic (PLEG): container finished" podID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerID="3096cc3b98256ba2c521a1e897218e89906e9bd536cf98ab0509fdf5bcce77c7" exitCode=60 Oct 29 23:30:04.388195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.387634 5736 scope.go:122] "RemoveContainer" containerID="3096cc3b98256ba2c521a1e897218e89906e9bd536cf98ab0509fdf5bcce77c7" Oct 29 23:30:04.388195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.387989 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" event={"ID":"95c61621-cfc4-48fb-ad57-b77f44df64fa","Type":"ContainerDied","Data":"3096cc3b98256ba2c521a1e897218e89906e9bd536cf98ab0509fdf5bcce77c7"} Oct 29 23:30:04.394090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.394062 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" event={"ID":"a7da5bb7-45be-4fc5-94d9-de6402481acb","Type":"ContainerStarted","Data":"b764b9e4700824597ccb0898457549fd489afc16c80884b71675a07290d547f5"} Oct 29 23:30:04.394406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:04.394388 5736 scope.go:122] "RemoveContainer" containerID="b764b9e4700824597ccb0898457549fd489afc16c80884b71675a07290d547f5" Oct 29 23:30:05.199606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.197396 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4"] Oct 29 23:30:05.199606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.198947 5736 cpu_manager.go:414] "containerMap: removing container" podUID="baa1e792-c7be-446e-850e-a645eb1f9df9" containerName="payload-processing" Oct 29 23:30:05.199606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.198972 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa1e792-c7be-446e-850e-a645eb1f9df9" containerName="payload-processing" Oct 29 23:30:05.199606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.199265 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa1e792-c7be-446e-850e-a645eb1f9df9" containerName="payload-processing" Oct 29 23:30:05.206921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.206765 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.225414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.223945 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4"] Oct 29 23:30:05.235697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.235670 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 29 23:30:05.235957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.235938 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 29 23:30:05.236069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.236018 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 29 23:30:05.236212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.236146 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-r8x84\"" Oct 29 23:30:05.236285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.236216 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 29 23:30:05.325900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.325865 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-520226f6ef3a\"" Oct 29 23:30:05.455562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.455524 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-provider-credentials\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.455859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.455830 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-plugins-config-volume\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.456009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.455994 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-routing-overlay-volume\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.456159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.456145 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkt9\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-kube-api-access-zbkt9\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.559078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.559033 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-provider-credentials\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.564738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.561089 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-plugins-config-volume\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.564738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.561157 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-routing-overlay-volume\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.564738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.561228 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkt9\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-kube-api-access-zbkt9\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.564738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.562862 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-plugins-config-volume\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.565781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.565759 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-routing-overlay-volume\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.754186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.749929 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-provider-credentials\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:05.958266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.952268 5736 generic.go:356] "Generic (PLEG): container finished" podID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerID="b764b9e4700824597ccb0898457549fd489afc16c80884b71675a07290d547f5" exitCode=60 Oct 29 23:30:05.958266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.952412 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" event={"ID":"a7da5bb7-45be-4fc5-94d9-de6402481acb","Type":"ContainerDied","Data":"b764b9e4700824597ccb0898457549fd489afc16c80884b71675a07290d547f5"} Oct 29 23:30:05.958266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.952494 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" event={"ID":"a7da5bb7-45be-4fc5-94d9-de6402481acb","Type":"ContainerStarted","Data":"f0515794027cc0815f0077c7e25f430e1fe6038b22c010e2e246b003b3ba5773"} Oct 29 23:30:06.007209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:05.993936 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkt9\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-kube-api-access-zbkt9\") pod \"payload-processing-external-model-77c9d69db9-rrsq4\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:06.022393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.022356 5736 generic.go:356] "Generic (PLEG): container finished" podID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerID="8eab7d400b770b6d7d340f323f2ff2d18610bc23a27aeb8e15b86066e730d981" exitCode=60 Oct 29 23:30:06.022544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.022457 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" event={"ID":"e1e77280-ca64-49f3-a6ca-ee1ec7573142","Type":"ContainerDied","Data":"8eab7d400b770b6d7d340f323f2ff2d18610bc23a27aeb8e15b86066e730d981"} Oct 29 23:30:06.022544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.022506 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" event={"ID":"e1e77280-ca64-49f3-a6ca-ee1ec7573142","Type":"ContainerStarted","Data":"e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418"} Oct 29 23:30:06.022544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.022535 5736 scope.go:122] "RemoveContainer" containerID="8eab7d400b770b6d7d340f323f2ff2d18610bc23a27aeb8e15b86066e730d981" Oct 29 23:30:06.023151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.023120 5736 scope.go:122] "RemoveContainer" containerID="e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418" Oct 29 23:30:06.023530 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:06.023500 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx_odh-ai-gateway-infra(e1e77280-ca64-49f3-a6ca-ee1ec7573142)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" Oct 29 23:30:06.027481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.027425 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" podStartSLOduration=6.027407868 podStartE2EDuration="6.027407868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:05.99462555 +0000 UTC m=+2803.110545158" watchObservedRunningTime="2026-10-29 23:30:06.027407868 +0000 UTC m=+2803.143327475" Oct 29 23:30:06.114013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.107097 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" event={"ID":"f76f9f99-0971-49bb-b819-23ce4f897001","Type":"ContainerStarted","Data":"daa0171f8cdec8525830ca5c9b345522ace850f2b3db0bc8b41eafbe47546a59"} Oct 29 23:30:06.225548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.223154 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:06.251167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.247176 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" event={"ID":"ffbe9920-f212-4ec5-8055-754d35f7e76e","Type":"ContainerStarted","Data":"e58de3c7d4e4915369f6d2cd28989d8d7710f535d509123b8f2d2a6c86cdb5fc"} Oct 29 23:30:06.277898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.277848 5736 generic.go:356] "Generic (PLEG): container finished" podID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerID="2b77db6dae48d3bc751e5cf61da95b470f0f7133998c35bfee54d0904d3c929f" exitCode=60 Oct 29 23:30:06.278275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.278233 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" event={"ID":"08d3c108-ba23-4291-a0cf-909c58d2a8bb","Type":"ContainerDied","Data":"2b77db6dae48d3bc751e5cf61da95b470f0f7133998c35bfee54d0904d3c929f"} Oct 29 23:30:06.320605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.295452 5736 scope.go:122] "RemoveContainer" containerID="e58de3c7d4e4915369f6d2cd28989d8d7710f535d509123b8f2d2a6c86cdb5fc" Oct 29 23:30:06.324745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.324700 5736 scope.go:122] "RemoveContainer" containerID="2b77db6dae48d3bc751e5cf61da95b470f0f7133998c35bfee54d0904d3c929f" Oct 29 23:30:06.344607 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:06.325241 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj_odh-ai-gateway-infra(08d3c108-ba23-4291-a0cf-909c58d2a8bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" Oct 29 23:30:06.344962 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:06.344928 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5_odh-ai-gateway-infra(ffbe9920-f212-4ec5-8055-754d35f7e76e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" Oct 29 23:30:06.442902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.436913 5736 generic.go:356] "Generic (PLEG): container finished" podID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerID="84fce66d7083c25acc8ba0904455a8272a0b8709e92b06457566f47af1c63ebc" exitCode=60 Oct 29 23:30:06.442902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.437024 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" event={"ID":"8aad335b-fbf2-41ff-b597-ce071f6837e6","Type":"ContainerDied","Data":"84fce66d7083c25acc8ba0904455a8272a0b8709e92b06457566f47af1c63ebc"} Oct 29 23:30:06.489050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.479701 5736 scope.go:122] "RemoveContainer" containerID="cff090a11be52bcd7e112ac8e79a8d0861cd2389095c62903f78b661c562dd1e" Oct 29 23:30:06.498376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.494827 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" event={"ID":"8d03a3d3-acce-41e7-97ff-47c0beb3a320","Type":"ContainerStarted","Data":"1c981cef7b9d9780864da4d25ba408e8228be69051b9175e13b5fe3a158994ae"} Oct 29 23:30:06.498376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.495340 5736 scope.go:122] "RemoveContainer" containerID="1c981cef7b9d9780864da4d25ba408e8228be69051b9175e13b5fe3a158994ae" Oct 29 23:30:06.498376 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:06.498266 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv_odh-ai-gateway-infra(8d03a3d3-acce-41e7-97ff-47c0beb3a320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" Oct 29 23:30:06.676200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.674062 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85c9c95994-drxzj"] Oct 29 23:30:06.682350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.682322 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.696020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.692356 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c9c95994-drxzj"] Oct 29 23:30:06.700898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.700866 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7471e9803417\"" Oct 29 23:30:06.723905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.723197 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-provider-credentials\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.723905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.723322 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhkt\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-kube-api-access-8mhkt\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.723905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.723481 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-plugins-config-volume\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.723905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.723624 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-routing-overlay-volume\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.825859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.825828 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-plugins-config-volume\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.826000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.825950 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-routing-overlay-volume\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.826116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.826101 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-provider-credentials\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.826192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.826176 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhkt\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-kube-api-access-8mhkt\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.826748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.826723 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-routing-overlay-volume\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.826866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.826822 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-plugins-config-volume\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.828533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.828514 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-provider-credentials\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.847525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.847495 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhkt\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-kube-api-access-8mhkt\") pod \"payload-processing-external-model-85c9c95994-drxzj\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:06.944046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:06.943991 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4"] Oct 29 23:30:06.971838 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:06.971807 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93942c3a_1475_4a0d_a9cc_10ec9256b4e6.slice/crio-9e9092e770c53a2901dd70d60f87ff1dace52bb8079e1da56fe1f4a42ac54254 WatchSource:0}: Error finding container 9e9092e770c53a2901dd70d60f87ff1dace52bb8079e1da56fe1f4a42ac54254: Status 404 returned error can't find the container with id 9e9092e770c53a2901dd70d60f87ff1dace52bb8079e1da56fe1f4a42ac54254 Oct 29 23:30:07.064152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.064073 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" podUID="7d262f4a-f03e-4ba9-92cb-8cc3207181a7" containerName="payload-processing" containerID="cri-o://ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546" gracePeriod=25 Oct 29 23:30:07.367329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.366048 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:07.452855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.452820 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:30:07.532124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.532037 5736 generic.go:356] "Generic (PLEG): container finished" podID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerID="50d9c2f3929decc58a0d456a21517c694dd7f4081d4d2d5ebb032d2ea271802c" exitCode=60 Oct 29 23:30:07.532413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.532390 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" event={"ID":"992c6925-45f5-4cb3-a436-f032da9b30f5","Type":"ContainerDied","Data":"50d9c2f3929decc58a0d456a21517c694dd7f4081d4d2d5ebb032d2ea271802c"} Oct 29 23:30:07.532652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.532626 5736 scope.go:122] "RemoveContainer" containerID="ce2a5b91c838b360ec9a2c6746ba1edad29635fe36811d0c3e7aa650927d383d" Oct 29 23:30:07.533438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.533119 5736 scope.go:122] "RemoveContainer" containerID="50d9c2f3929decc58a0d456a21517c694dd7f4081d4d2d5ebb032d2ea271802c" Oct 29 23:30:07.533752 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.533664 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2_odh-ai-gateway-infra(992c6925-45f5-4cb3-a436-f032da9b30f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" Oct 29 23:30:07.536897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.536875 5736 generic.go:356] "Generic (PLEG): container finished" podID="48af7825-6733-4565-8347-58c931a08499" containerID="3b8c521790e16ed217b2c00dadc1f3af57f2b9cf6b44a2c09d9b98d1b6de06a5" exitCode=60 Oct 29 23:30:07.537072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.536932 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" event={"ID":"48af7825-6733-4565-8347-58c931a08499","Type":"ContainerDied","Data":"3b8c521790e16ed217b2c00dadc1f3af57f2b9cf6b44a2c09d9b98d1b6de06a5"} Oct 29 23:30:07.537302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.537281 5736 scope.go:122] "RemoveContainer" containerID="3b8c521790e16ed217b2c00dadc1f3af57f2b9cf6b44a2c09d9b98d1b6de06a5" Oct 29 23:30:07.537710 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.537680 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-qfk7d_odh-ai-gateway-infra(48af7825-6733-4565-8347-58c931a08499)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" podUID="48af7825-6733-4565-8347-58c931a08499" Oct 29 23:30:07.539659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.539634 5736 generic.go:356] "Generic (PLEG): container finished" podID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerID="4e0f5cb2a117c96335fd1286ad453a1f38f46d0f19a9d5e49b50cda00fe053b6" exitCode=60 Oct 29 23:30:07.539881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.539724 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" event={"ID":"c19f4669-9c01-4e64-a396-b755fdbfddd6","Type":"ContainerDied","Data":"4e0f5cb2a117c96335fd1286ad453a1f38f46d0f19a9d5e49b50cda00fe053b6"} Oct 29 23:30:07.540190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.540166 5736 scope.go:122] "RemoveContainer" containerID="4e0f5cb2a117c96335fd1286ad453a1f38f46d0f19a9d5e49b50cda00fe053b6" Oct 29 23:30:07.540641 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.540604 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2_odh-ai-gateway-infra(c19f4669-9c01-4e64-a396-b755fdbfddd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" Oct 29 23:30:07.542567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.542541 5736 generic.go:356] "Generic (PLEG): container finished" podID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerID="8ad5fcd980b629a816766b38648204b0358c80dde9e6ee9763fce525a60e3ccb" exitCode=60 Oct 29 23:30:07.542703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.542635 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" event={"ID":"46519c4c-5baa-453c-8e06-d2c1e41eb6dd","Type":"ContainerDied","Data":"8ad5fcd980b629a816766b38648204b0358c80dde9e6ee9763fce525a60e3ccb"} Oct 29 23:30:07.543053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.543030 5736 scope.go:122] "RemoveContainer" containerID="8ad5fcd980b629a816766b38648204b0358c80dde9e6ee9763fce525a60e3ccb" Oct 29 23:30:07.543390 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.543365 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb_odh-ai-gateway-infra(46519c4c-5baa-453c-8e06-d2c1e41eb6dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" Oct 29 23:30:07.545627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.545602 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" event={"ID":"93942c3a-1475-4a0d-a9cc-10ec9256b4e6","Type":"ContainerStarted","Data":"8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a"} Oct 29 23:30:07.545778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.545759 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" event={"ID":"93942c3a-1475-4a0d-a9cc-10ec9256b4e6","Type":"ContainerStarted","Data":"9e9092e770c53a2901dd70d60f87ff1dace52bb8079e1da56fe1f4a42ac54254"} Oct 29 23:30:07.546081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.546045 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:07.548320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.548299 5736 generic.go:356] "Generic (PLEG): container finished" podID="7d262f4a-f03e-4ba9-92cb-8cc3207181a7" containerID="ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546" exitCode=0 Oct 29 23:30:07.548434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.548409 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" event={"ID":"7d262f4a-f03e-4ba9-92cb-8cc3207181a7","Type":"ContainerDied","Data":"ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546"} Oct 29 23:30:07.548515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.548476 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" event={"ID":"7d262f4a-f03e-4ba9-92cb-8cc3207181a7","Type":"ContainerDied","Data":"c8dcec4ddd5cc24df2f762c2ed112cfddba6e4fda16f3cd3bea116bceb5d8083"} Oct 29 23:30:07.548657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.548626 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t" Oct 29 23:30:07.554134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.551433 5736 generic.go:356] "Generic (PLEG): container finished" podID="470d3dc9-0310-418e-be47-56549cb7573d" containerID="9f56a276b93a621ae05a97aece529d63e48f10199cfc4f999bd2fa473b670a8d" exitCode=60 Oct 29 23:30:07.554134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.551517 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" event={"ID":"470d3dc9-0310-418e-be47-56549cb7573d","Type":"ContainerDied","Data":"9f56a276b93a621ae05a97aece529d63e48f10199cfc4f999bd2fa473b670a8d"} Oct 29 23:30:07.554134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.551916 5736 scope.go:122] "RemoveContainer" containerID="9f56a276b93a621ae05a97aece529d63e48f10199cfc4f999bd2fa473b670a8d" Oct 29 23:30:07.554134 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.552208 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7_odh-ai-gateway-infra(470d3dc9-0310-418e-be47-56549cb7573d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" podUID="470d3dc9-0310-418e-be47-56549cb7573d" Oct 29 23:30:07.554480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.554179 5736 scope.go:122] "RemoveContainer" containerID="446a7f5f406c6933d4db22fbb97ab013dce99ba9f643c12affdc3fc7c9b35f90" Oct 29 23:30:07.554796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.554770 5736 generic.go:356] "Generic (PLEG): container finished" podID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerID="450661a923286541c0a4b75a6641c1e29141ffe100db78f6e5b141508ec89cf9" exitCode=60 Oct 29 23:30:07.554894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.554840 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" event={"ID":"ca0cab44-e6f4-44b3-be68-2e14816aeb27","Type":"ContainerDied","Data":"450661a923286541c0a4b75a6641c1e29141ffe100db78f6e5b141508ec89cf9"} Oct 29 23:30:07.555204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.555182 5736 scope.go:122] "RemoveContainer" containerID="450661a923286541c0a4b75a6641c1e29141ffe100db78f6e5b141508ec89cf9" Oct 29 23:30:07.555487 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.555462 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd_odh-ai-gateway-infra(ca0cab44-e6f4-44b3-be68-2e14816aeb27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" Oct 29 23:30:07.558214 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.558190 5736 generic.go:356] "Generic (PLEG): container finished" podID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerID="241d734b69918dd57ccc90f5d6d7664aabaa25e059db002689c13c6ba0d4a8cb" exitCode=60 Oct 29 23:30:07.558321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.558288 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" event={"ID":"95c61621-cfc4-48fb-ad57-b77f44df64fa","Type":"ContainerDied","Data":"241d734b69918dd57ccc90f5d6d7664aabaa25e059db002689c13c6ba0d4a8cb"} Oct 29 23:30:07.558527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.558508 5736 scope.go:122] "RemoveContainer" containerID="241d734b69918dd57ccc90f5d6d7664aabaa25e059db002689c13c6ba0d4a8cb" Oct 29 23:30:07.558820 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.558798 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7_odh-ai-gateway-infra(95c61621-cfc4-48fb-ad57-b77f44df64fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" Oct 29 23:30:07.560748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.560723 5736 generic.go:356] "Generic (PLEG): container finished" podID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerID="f0515794027cc0815f0077c7e25f430e1fe6038b22c010e2e246b003b3ba5773" exitCode=60 Oct 29 23:30:07.560852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.560816 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" event={"ID":"a7da5bb7-45be-4fc5-94d9-de6402481acb","Type":"ContainerDied","Data":"f0515794027cc0815f0077c7e25f430e1fe6038b22c010e2e246b003b3ba5773"} Oct 29 23:30:07.561216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.561198 5736 scope.go:122] "RemoveContainer" containerID="f0515794027cc0815f0077c7e25f430e1fe6038b22c010e2e246b003b3ba5773" Oct 29 23:30:07.561615 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.561554 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx_odh-ai-gateway-infra(a7da5bb7-45be-4fc5-94d9-de6402481acb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" Oct 29 23:30:07.562927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.562903 5736 generic.go:356] "Generic (PLEG): container finished" podID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerID="e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418" exitCode=60 Oct 29 23:30:07.563021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.562974 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" event={"ID":"e1e77280-ca64-49f3-a6ca-ee1ec7573142","Type":"ContainerDied","Data":"e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418"} Oct 29 23:30:07.563336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.563314 5736 scope.go:122] "RemoveContainer" containerID="e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418" Oct 29 23:30:07.563641 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.563575 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx_odh-ai-gateway-infra(e1e77280-ca64-49f3-a6ca-ee1ec7573142)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" Oct 29 23:30:07.564553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.564526 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-kube-api-access-xvbhz\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-kube-api-access-xvbhz\") pod \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " Oct 29 23:30:07.564741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.564635 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-plugins-config-volume\") pod \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " Oct 29 23:30:07.564741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.564687 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-provider-credentials\") pod \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " Oct 29 23:30:07.564860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.564798 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-routing-overlay-volume\") pod \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\" (UID: \"7d262f4a-f03e-4ba9-92cb-8cc3207181a7\") " Oct 29 23:30:07.565332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.565305 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-plugins-config-volume" pod "7d262f4a-f03e-4ba9-92cb-8cc3207181a7" (UID: "7d262f4a-f03e-4ba9-92cb-8cc3207181a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:07.567262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.566442 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-routing-overlay-volume" pod "7d262f4a-f03e-4ba9-92cb-8cc3207181a7" (UID: "7d262f4a-f03e-4ba9-92cb-8cc3207181a7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:07.567262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.566750 5736 generic.go:356] "Generic (PLEG): container finished" podID="f76f9f99-0971-49bb-b819-23ce4f897001" containerID="daa0171f8cdec8525830ca5c9b345522ace850f2b3db0bc8b41eafbe47546a59" exitCode=60 Oct 29 23:30:07.567262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.566819 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" event={"ID":"f76f9f99-0971-49bb-b819-23ce4f897001","Type":"ContainerDied","Data":"daa0171f8cdec8525830ca5c9b345522ace850f2b3db0bc8b41eafbe47546a59"} Oct 29 23:30:07.567262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.567049 5736 scope.go:122] "RemoveContainer" containerID="daa0171f8cdec8525830ca5c9b345522ace850f2b3db0bc8b41eafbe47546a59" Oct 29 23:30:07.567262 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.567227 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb_odh-ai-gateway-infra(f76f9f99-0971-49bb-b819-23ce4f897001)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" Oct 29 23:30:07.568286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.568258 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-kube-api-access-xvbhz" pod "7d262f4a-f03e-4ba9-92cb-8cc3207181a7" (UID: "7d262f4a-f03e-4ba9-92cb-8cc3207181a7"). InnerVolumeSpecName "kube-api-access-xvbhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:07.568387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.568365 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-provider-credentials" pod "7d262f4a-f03e-4ba9-92cb-8cc3207181a7" (UID: "7d262f4a-f03e-4ba9-92cb-8cc3207181a7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:07.569870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.569847 5736 generic.go:356] "Generic (PLEG): container finished" podID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerID="e58de3c7d4e4915369f6d2cd28989d8d7710f535d509123b8f2d2a6c86cdb5fc" exitCode=60 Oct 29 23:30:07.569954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.569918 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" event={"ID":"ffbe9920-f212-4ec5-8055-754d35f7e76e","Type":"ContainerDied","Data":"e58de3c7d4e4915369f6d2cd28989d8d7710f535d509123b8f2d2a6c86cdb5fc"} Oct 29 23:30:07.570209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.570185 5736 scope.go:122] "RemoveContainer" containerID="e58de3c7d4e4915369f6d2cd28989d8d7710f535d509123b8f2d2a6c86cdb5fc" Oct 29 23:30:07.570469 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.570440 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5_odh-ai-gateway-infra(ffbe9920-f212-4ec5-8055-754d35f7e76e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" Oct 29 23:30:07.572898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.572875 5736 scope.go:122] "RemoveContainer" containerID="2b77db6dae48d3bc751e5cf61da95b470f0f7133998c35bfee54d0904d3c929f" Oct 29 23:30:07.573709 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.573175 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj_odh-ai-gateway-infra(08d3c108-ba23-4291-a0cf-909c58d2a8bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" Oct 29 23:30:07.573996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.573976 5736 scope.go:122] "RemoveContainer" containerID="60eeaa47e85eb02c6d05ca3d91ef00fce7c0617057bee69e59dbd6b2dab6e237" Oct 29 23:30:07.575358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.575339 5736 generic.go:356] "Generic (PLEG): container finished" podID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerID="9fb94e26f7937769e1a2066bb39cc836b26ca1497fe20f7c1b310a8a390d52c4" exitCode=60 Oct 29 23:30:07.575454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.575426 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" event={"ID":"8aad335b-fbf2-41ff-b597-ce071f6837e6","Type":"ContainerDied","Data":"9fb94e26f7937769e1a2066bb39cc836b26ca1497fe20f7c1b310a8a390d52c4"} Oct 29 23:30:07.575682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.575661 5736 scope.go:122] "RemoveContainer" containerID="9fb94e26f7937769e1a2066bb39cc836b26ca1497fe20f7c1b310a8a390d52c4" Oct 29 23:30:07.575956 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.575929 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh_odh-ai-gateway-infra(8aad335b-fbf2-41ff-b597-ce071f6837e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" Oct 29 23:30:07.577665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.577640 5736 generic.go:356] "Generic (PLEG): container finished" podID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerID="1c981cef7b9d9780864da4d25ba408e8228be69051b9175e13b5fe3a158994ae" exitCode=60 Oct 29 23:30:07.577780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.577696 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" event={"ID":"8d03a3d3-acce-41e7-97ff-47c0beb3a320","Type":"ContainerDied","Data":"1c981cef7b9d9780864da4d25ba408e8228be69051b9175e13b5fe3a158994ae"} Oct 29 23:30:07.577936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.577910 5736 scope.go:122] "RemoveContainer" containerID="1c981cef7b9d9780864da4d25ba408e8228be69051b9175e13b5fe3a158994ae" Oct 29 23:30:07.578199 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.578176 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv_odh-ai-gateway-infra(8d03a3d3-acce-41e7-97ff-47c0beb3a320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" Oct 29 23:30:07.580268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.580244 5736 generic.go:356] "Generic (PLEG): container finished" podID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerID="dee99d6fba2b76ea250548deba0af6224bacc2fb6223c9e3dd3240c3d0e25567" exitCode=60 Oct 29 23:30:07.580508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.580321 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" event={"ID":"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2","Type":"ContainerDied","Data":"dee99d6fba2b76ea250548deba0af6224bacc2fb6223c9e3dd3240c3d0e25567"} Oct 29 23:30:07.580688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.580538 5736 scope.go:122] "RemoveContainer" containerID="dee99d6fba2b76ea250548deba0af6224bacc2fb6223c9e3dd3240c3d0e25567" Oct 29 23:30:07.580857 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.580827 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw_odh-ai-gateway-infra(21bb330a-9e1a-4f01-ab2f-aabd1c199eb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" Oct 29 23:30:07.582441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.582419 5736 generic.go:356] "Generic (PLEG): container finished" podID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerID="1c0dc01b2a243320e78b5a6643a6824e1e2b243d4212cb9f05e8847a2a7f2806" exitCode=60 Oct 29 23:30:07.582533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.582498 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" event={"ID":"b8868c3b-f832-4b4c-8968-139096f19a4a","Type":"ContainerDied","Data":"1c0dc01b2a243320e78b5a6643a6824e1e2b243d4212cb9f05e8847a2a7f2806"} Oct 29 23:30:07.582711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.582690 5736 scope.go:122] "RemoveContainer" containerID="1c0dc01b2a243320e78b5a6643a6824e1e2b243d4212cb9f05e8847a2a7f2806" Oct 29 23:30:07.582981 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.582958 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n_odh-ai-gateway-infra(b8868c3b-f832-4b4c-8968-139096f19a4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" Oct 29 23:30:07.584548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.584527 5736 generic.go:356] "Generic (PLEG): container finished" podID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerID="6aa359e050c3f8183cff0ead51097c3d71192c9360ca544517d6fcb0861ed380" exitCode=60 Oct 29 23:30:07.584668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.584609 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" event={"ID":"2b998c5e-6c2a-4065-898e-e4ba61b45a0a","Type":"ContainerDied","Data":"6aa359e050c3f8183cff0ead51097c3d71192c9360ca544517d6fcb0861ed380"} Oct 29 23:30:07.584899 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.584880 5736 scope.go:122] "RemoveContainer" containerID="6aa359e050c3f8183cff0ead51097c3d71192c9360ca544517d6fcb0861ed380" Oct 29 23:30:07.585169 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.585141 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp_odh-ai-gateway-infra(2b998c5e-6c2a-4065-898e-e4ba61b45a0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" Oct 29 23:30:07.586843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.586821 5736 generic.go:356] "Generic (PLEG): container finished" podID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerID="027c5750cb0b893676f24e16e7b5db9d69e418c502005a5af7baa8c8e00b0c71" exitCode=60 Oct 29 23:30:07.586926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.586889 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" event={"ID":"2ab2f333-8e37-4019-ae18-e4a748837c22","Type":"ContainerDied","Data":"027c5750cb0b893676f24e16e7b5db9d69e418c502005a5af7baa8c8e00b0c71"} Oct 29 23:30:07.587070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.587056 5736 scope.go:122] "RemoveContainer" containerID="027c5750cb0b893676f24e16e7b5db9d69e418c502005a5af7baa8c8e00b0c71" Oct 29 23:30:07.587268 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.587246 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb_odh-ai-gateway-infra(2ab2f333-8e37-4019-ae18-e4a748837c22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" Oct 29 23:30:07.591650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.591626 5736 scope.go:122] "RemoveContainer" containerID="4169995f14e493b0d6e2844ad78e1b2ea9450c46b7dce4c306729a91eab9ce98" Oct 29 23:30:07.607383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.607366 5736 scope.go:122] "RemoveContainer" containerID="ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546" Oct 29 23:30:07.644011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.643985 5736 scope.go:122] "RemoveContainer" containerID="ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546" Oct 29 23:30:07.644485 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:07.644453 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546\": container with ID starting with ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546 not found: ID does not exist" containerID="ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546" Oct 29 23:30:07.644640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.644489 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546"} err="failed to get container status \"ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546\": rpc error: code = NotFound desc = could not find container \"ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546\": container with ID starting with ff89be4e3f35ad571cb91df442b0e851f512efd6594f2fc484b56fb1fcdc2546 not found: ID does not exist" Oct 29 23:30:07.644640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.644508 5736 scope.go:122] "RemoveContainer" containerID="945b8924ecbde68eaf4c78567bf2362689615d6e8766735033f63fa5084feb29" Oct 29 23:30:07.660114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.660088 5736 scope.go:122] "RemoveContainer" containerID="1d9abccbf00aed42f600d28052f1ec5c7f4d91faac1fac60be5f0151289e9243" Oct 29 23:30:07.669995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.669886 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbhz\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-kube-api-access-xvbhz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:07.669995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.669915 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:07.669995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.669955 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:07.669995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.669973 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d262f4a-f03e-4ba9-92cb-8cc3207181a7-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:07.679535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.679510 5736 scope.go:122] "RemoveContainer" containerID="3096cc3b98256ba2c521a1e897218e89906e9bd536cf98ab0509fdf5bcce77c7" Oct 29 23:30:07.694002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.693982 5736 scope.go:122] "RemoveContainer" containerID="b764b9e4700824597ccb0898457549fd489afc16c80884b71675a07290d547f5" Oct 29 23:30:07.701247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.701202 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" podStartSLOduration=2.701188772 podStartE2EDuration="2.701188772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:07.698946815 +0000 UTC m=+2804.814866422" watchObservedRunningTime="2026-10-29 23:30:07.701188772 +0000 UTC m=+2804.817108356" Oct 29 23:30:07.710769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.710746 5736 scope.go:122] "RemoveContainer" containerID="ea4edd537dda8fc9beb03ca34ce180479681712b55ea7f19d1808c4ba0267347" Oct 29 23:30:07.724840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.724823 5736 scope.go:122] "RemoveContainer" containerID="f6f078496544220063d49f43246d715160c20b6845f07b77c8dfc7460a4c028c" Oct 29 23:30:07.730734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.730709 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c9c95994-drxzj"] Oct 29 23:30:07.731740 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:07.731713 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce03d100_f93d_461f_8d8e_c80639f5a2b9.slice/crio-196f5a75db80387eb63f55d0b909c7ce1eebe507d6a636beb9d27c296b9ee914 WatchSource:0}: Error finding container 196f5a75db80387eb63f55d0b909c7ce1eebe507d6a636beb9d27c296b9ee914: Status 404 returned error can't find the container with id 196f5a75db80387eb63f55d0b909c7ce1eebe507d6a636beb9d27c296b9ee914 Oct 29 23:30:07.751864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.751837 5736 scope.go:122] "RemoveContainer" containerID="84fce66d7083c25acc8ba0904455a8272a0b8709e92b06457566f47af1c63ebc" Oct 29 23:30:07.780504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.780481 5736 scope.go:122] "RemoveContainer" containerID="6075f26a474032745bef9ad2eaf14a10052dabebb4a9d7a92f390ce8188901ab" Oct 29 23:30:07.797801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.797780 5736 scope.go:122] "RemoveContainer" containerID="fb1352e8cf33732c290240df9280e374d6df6c480260e5c103a2ead2fe4a5c95" Oct 29 23:30:07.816475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.816447 5736 scope.go:122] "RemoveContainer" containerID="992eedcaa5178a6eabcae206d9a76d765cae656c7a080cf0692d70266793854f" Oct 29 23:30:07.832330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.831462 5736 scope.go:122] "RemoveContainer" containerID="abca25941bfc5b7e55c83d6c0a758647af8401fc3ed96d3bf798e01eca26ed61" Oct 29 23:30:07.849812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:07.849788 5736 scope.go:122] "RemoveContainer" containerID="b4eb02c3ca32447447f6987024e1f7642220920360b98f3260309faf7740fa6a" Oct 29 23:30:08.405474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.405436 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t"] Oct 29 23:30:08.442669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.442626 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e10975/payload-processing-external-model-e2e-extmod-a-e10975-7ffckzw2t"] Oct 29 23:30:08.606706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.606675 5736 scope.go:122] "RemoveContainer" containerID="9f56a276b93a621ae05a97aece529d63e48f10199cfc4f999bd2fa473b670a8d" Oct 29 23:30:08.607080 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.607051 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7_odh-ai-gateway-infra(470d3dc9-0310-418e-be47-56549cb7573d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" podUID="470d3dc9-0310-418e-be47-56549cb7573d" Oct 29 23:30:08.608452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.608421 5736 scope.go:122] "RemoveContainer" containerID="450661a923286541c0a4b75a6641c1e29141ffe100db78f6e5b141508ec89cf9" Oct 29 23:30:08.608868 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.608839 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd_odh-ai-gateway-infra(ca0cab44-e6f4-44b3-be68-2e14816aeb27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" Oct 29 23:30:08.610302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.610281 5736 scope.go:122] "RemoveContainer" containerID="241d734b69918dd57ccc90f5d6d7664aabaa25e059db002689c13c6ba0d4a8cb" Oct 29 23:30:08.610702 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.610665 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7_odh-ai-gateway-infra(95c61621-cfc4-48fb-ad57-b77f44df64fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" Oct 29 23:30:08.612219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.612190 5736 scope.go:122] "RemoveContainer" containerID="f0515794027cc0815f0077c7e25f430e1fe6038b22c010e2e246b003b3ba5773" Oct 29 23:30:08.612487 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.612465 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx_odh-ai-gateway-infra(a7da5bb7-45be-4fc5-94d9-de6402481acb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" Oct 29 23:30:08.614101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.614082 5736 scope.go:122] "RemoveContainer" containerID="daa0171f8cdec8525830ca5c9b345522ace850f2b3db0bc8b41eafbe47546a59" Oct 29 23:30:08.614407 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.614383 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb_odh-ai-gateway-infra(f76f9f99-0971-49bb-b819-23ce4f897001)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" Oct 29 23:30:08.615705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.615685 5736 scope.go:122] "RemoveContainer" containerID="e58de3c7d4e4915369f6d2cd28989d8d7710f535d509123b8f2d2a6c86cdb5fc" Oct 29 23:30:08.615953 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.615927 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5_odh-ai-gateway-infra(ffbe9920-f212-4ec5-8055-754d35f7e76e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" Oct 29 23:30:08.617094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.617075 5736 scope.go:122] "RemoveContainer" containerID="9fb94e26f7937769e1a2066bb39cc836b26ca1497fe20f7c1b310a8a390d52c4" Oct 29 23:30:08.617348 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.617326 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh_odh-ai-gateway-infra(8aad335b-fbf2-41ff-b597-ce071f6837e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" Oct 29 23:30:08.618733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.618710 5736 scope.go:122] "RemoveContainer" containerID="1c981cef7b9d9780864da4d25ba408e8228be69051b9175e13b5fe3a158994ae" Oct 29 23:30:08.618969 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.618947 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv_odh-ai-gateway-infra(8d03a3d3-acce-41e7-97ff-47c0beb3a320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" Oct 29 23:30:08.620314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.620295 5736 scope.go:122] "RemoveContainer" containerID="dee99d6fba2b76ea250548deba0af6224bacc2fb6223c9e3dd3240c3d0e25567" Oct 29 23:30:08.620546 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.620526 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw_odh-ai-gateway-infra(21bb330a-9e1a-4f01-ab2f-aabd1c199eb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" Oct 29 23:30:08.621854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.621837 5736 scope.go:122] "RemoveContainer" containerID="1c0dc01b2a243320e78b5a6643a6824e1e2b243d4212cb9f05e8847a2a7f2806" Oct 29 23:30:08.622091 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.622072 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n_odh-ai-gateway-infra(b8868c3b-f832-4b4c-8968-139096f19a4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" Oct 29 23:30:08.623304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.623273 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" event={"ID":"ce03d100-f93d-461f-8d8e-c80639f5a2b9","Type":"ContainerStarted","Data":"c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a"} Oct 29 23:30:08.623401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.623325 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" event={"ID":"ce03d100-f93d-461f-8d8e-c80639f5a2b9","Type":"ContainerStarted","Data":"196f5a75db80387eb63f55d0b909c7ce1eebe507d6a636beb9d27c296b9ee914"} Oct 29 23:30:08.623401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.623378 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:08.625767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.625731 5736 scope.go:122] "RemoveContainer" containerID="6aa359e050c3f8183cff0ead51097c3d71192c9360ca544517d6fcb0861ed380" Oct 29 23:30:08.626016 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.625983 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp_odh-ai-gateway-infra(2b998c5e-6c2a-4065-898e-e4ba61b45a0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" Oct 29 23:30:08.627527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.627509 5736 scope.go:122] "RemoveContainer" containerID="027c5750cb0b893676f24e16e7b5db9d69e418c502005a5af7baa8c8e00b0c71" Oct 29 23:30:08.627788 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.627762 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb_odh-ai-gateway-infra(2ab2f333-8e37-4019-ae18-e4a748837c22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" Oct 29 23:30:08.630092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.630073 5736 scope.go:122] "RemoveContainer" containerID="50d9c2f3929decc58a0d456a21517c694dd7f4081d4d2d5ebb032d2ea271802c" Oct 29 23:30:08.630326 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.630286 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2_odh-ai-gateway-infra(992c6925-45f5-4cb3-a436-f032da9b30f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" Oct 29 23:30:08.632471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.632451 5736 scope.go:122] "RemoveContainer" containerID="3b8c521790e16ed217b2c00dadc1f3af57f2b9cf6b44a2c09d9b98d1b6de06a5" Oct 29 23:30:08.632747 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.632724 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-qfk7d_odh-ai-gateway-infra(48af7825-6733-4565-8347-58c931a08499)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" podUID="48af7825-6733-4565-8347-58c931a08499" Oct 29 23:30:08.634235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.634217 5736 scope.go:122] "RemoveContainer" containerID="4e0f5cb2a117c96335fd1286ad453a1f38f46d0f19a9d5e49b50cda00fe053b6" Oct 29 23:30:08.634440 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.634417 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2_odh-ai-gateway-infra(c19f4669-9c01-4e64-a396-b755fdbfddd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" Oct 29 23:30:08.636373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.636347 5736 scope.go:122] "RemoveContainer" containerID="8ad5fcd980b629a816766b38648204b0358c80dde9e6ee9763fce525a60e3ccb" Oct 29 23:30:08.636987 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.636958 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb_odh-ai-gateway-infra(46519c4c-5baa-453c-8e06-d2c1e41eb6dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" Oct 29 23:30:08.637787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:08.637763 5736 scope.go:122] "RemoveContainer" containerID="e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418" Oct 29 23:30:08.638094 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:08.638065 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx_odh-ai-gateway-infra(e1e77280-ca64-49f3-a6ca-ee1ec7573142)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" Oct 29 23:30:09.007409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:09.007374 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d262f4a-f03e-4ba9-92cb-8cc3207181a7" path="/var/lib/kubelet/pods/7d262f4a-f03e-4ba9-92cb-8cc3207181a7/volumes" Oct 29 23:30:09.139731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:09.139677 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" podStartSLOduration=3.139661665 podStartE2EDuration="3.139661665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:09.136351271 +0000 UTC m=+2806.252270877" watchObservedRunningTime="2026-10-29 23:30:09.139661665 +0000 UTC m=+2806.255581273" Oct 29 23:30:14.168614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.168564 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx"] Oct 29 23:30:14.169694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.169670 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7d262f4a-f03e-4ba9-92cb-8cc3207181a7" containerName="payload-processing" Oct 29 23:30:14.169694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.169694 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d262f4a-f03e-4ba9-92cb-8cc3207181a7" containerName="payload-processing" Oct 29 23:30:14.169886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.169861 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d262f4a-f03e-4ba9-92cb-8cc3207181a7" containerName="payload-processing" Oct 29 23:30:14.175345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.175323 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:14.188738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.188715 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-eba670\"/\"openshift-service-ca.crt\"" Oct 29 23:30:14.191717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.191694 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx"] Oct 29 23:30:14.197199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.197170 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-eba670\"/\"default-dockercfg-ngt9w\"" Oct 29 23:30:14.199366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.199347 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-eba670\"/\"e2e-usage-sim-18d73f-cert\"" Oct 29 23:30:14.199366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.199355 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-eba670\"/\"kube-root-ca.crt\"" Oct 29 23:30:14.292639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.292600 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh5b6\" (UniqueName: \"kubernetes.io/projected/7292011e-66f4-46ee-b0b0-6b023e570dbe-kube-api-access-mh5b6\") pod \"e2e-usage-sim-18d73f-95c45d9f-9fmcx\" (UID: \"7292011e-66f4-46ee-b0b0-6b023e570dbe\") " pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:14.292851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.292827 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7292011e-66f4-46ee-b0b0-6b023e570dbe-tls\") pod \"e2e-usage-sim-18d73f-95c45d9f-9fmcx\" (UID: \"7292011e-66f4-46ee-b0b0-6b023e570dbe\") " pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:14.395095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.395055 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh5b6\" (UniqueName: \"kubernetes.io/projected/7292011e-66f4-46ee-b0b0-6b023e570dbe-kube-api-access-mh5b6\") pod \"e2e-usage-sim-18d73f-95c45d9f-9fmcx\" (UID: \"7292011e-66f4-46ee-b0b0-6b023e570dbe\") " pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:14.395400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.395237 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7292011e-66f4-46ee-b0b0-6b023e570dbe-tls\") pod \"e2e-usage-sim-18d73f-95c45d9f-9fmcx\" (UID: \"7292011e-66f4-46ee-b0b0-6b023e570dbe\") " pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:14.403845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.403816 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7292011e-66f4-46ee-b0b0-6b023e570dbe-tls\") pod \"e2e-usage-sim-18d73f-95c45d9f-9fmcx\" (UID: \"7292011e-66f4-46ee-b0b0-6b023e570dbe\") " pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:14.417689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.417657 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh5b6\" (UniqueName: \"kubernetes.io/projected/7292011e-66f4-46ee-b0b0-6b023e570dbe-kube-api-access-mh5b6\") pod \"e2e-usage-sim-18d73f-95c45d9f-9fmcx\" (UID: \"7292011e-66f4-46ee-b0b0-6b023e570dbe\") " pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:14.488961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.488925 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:14.803952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:14.803922 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx"] Oct 29 23:30:14.804919 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:14.804889 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7292011e_66f4_46ee_b0b0_6b023e570dbe.slice/crio-a649d9ad90bbacde6f60a6191b75036e108e6149b5c1c2bda39544b32f324b2c WatchSource:0}: Error finding container a649d9ad90bbacde6f60a6191b75036e108e6149b5c1c2bda39544b32f324b2c: Status 404 returned error can't find the container with id a649d9ad90bbacde6f60a6191b75036e108e6149b5c1c2bda39544b32f324b2c Oct 29 23:30:15.753834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:15.753783 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" event={"ID":"7292011e-66f4-46ee-b0b0-6b023e570dbe","Type":"ContainerStarted","Data":"e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f"} Oct 29 23:30:15.754403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:15.754381 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:15.754526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:15.754510 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" event={"ID":"7292011e-66f4-46ee-b0b0-6b023e570dbe","Type":"ContainerStarted","Data":"a649d9ad90bbacde6f60a6191b75036e108e6149b5c1c2bda39544b32f324b2c"} Oct 29 23:30:15.782602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:15.777914 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:30:15.803786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:15.803722 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" podStartSLOduration=1.80370298 podStartE2EDuration="1.80370298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:15.795795241 +0000 UTC m=+2812.911714848" watchObservedRunningTime="2026-10-29 23:30:15.80370298 +0000 UTC m=+2812.919622587" Oct 29 23:30:16.996403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:16.996363 5736 scope.go:122] "RemoveContainer" containerID="241d734b69918dd57ccc90f5d6d7664aabaa25e059db002689c13c6ba0d4a8cb" Oct 29 23:30:16.996944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:16.996789 5736 scope.go:122] "RemoveContainer" containerID="8ad5fcd980b629a816766b38648204b0358c80dde9e6ee9763fce525a60e3ccb" Oct 29 23:30:16.997019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:16.996957 5736 scope.go:122] "RemoveContainer" containerID="2b77db6dae48d3bc751e5cf61da95b470f0f7133998c35bfee54d0904d3c929f" Oct 29 23:30:16.997533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:16.997171 5736 scope.go:122] "RemoveContainer" containerID="1c981cef7b9d9780864da4d25ba408e8228be69051b9175e13b5fe3a158994ae" Oct 29 23:30:17.146613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.146569 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4"] Oct 29 23:30:17.150476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.150452 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:17.186410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.186378 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9"] Oct 29 23:30:17.193593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.193558 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.207234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.207205 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-65d797949c01\"" Oct 29 23:30:17.212614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.212549 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9"] Oct 29 23:30:17.284483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.284453 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvm6\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-kube-api-access-hxvm6\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.284677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.284508 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-provider-credentials\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.284677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.284650 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-routing-overlay-volume\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.284846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.284830 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-plugins-config-volume\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.389506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.389453 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvm6\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-kube-api-access-hxvm6\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.389736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.389568 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-provider-credentials\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.389736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.389650 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-routing-overlay-volume\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.389905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.389886 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-plugins-config-volume\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.392569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.391012 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-plugins-config-volume\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.413631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.413350 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-routing-overlay-volume\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.431343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.431263 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvm6\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-kube-api-access-hxvm6\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.445456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.445419 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-provider-credentials\") pod \"payload-processing-external-model-7ddc8dbdbf-r8jd9\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.510801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.510721 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:17.776060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.775975 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" event={"ID":"46519c4c-5baa-453c-8e06-d2c1e41eb6dd","Type":"ContainerStarted","Data":"d3d7906c82869d2fb65ca12932ed8c8ed0ff779c7cda23e79235891bf8c2f241"} Oct 29 23:30:17.778042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.778013 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" event={"ID":"95c61621-cfc4-48fb-ad57-b77f44df64fa","Type":"ContainerStarted","Data":"272148c246284a76cbfb9e686dca93525e2f40a4827e56e15b3eb10db2012ce8"} Oct 29 23:30:17.780054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.780031 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" event={"ID":"08d3c108-ba23-4291-a0cf-909c58d2a8bb","Type":"ContainerStarted","Data":"5f4a42a2f287eb259b14c72bc800a6f836417d9dc688a1dd0bda28c0db04a8d5"} Oct 29 23:30:17.782167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.782142 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" event={"ID":"8d03a3d3-acce-41e7-97ff-47c0beb3a320","Type":"ContainerStarted","Data":"0b517256da627703a20b2bc0fc956e507714527d1cd4cba806aed814d1c94d92"} Oct 29 23:30:17.819854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.819790 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" podStartSLOduration=17.819775828 podStartE2EDuration="17.819775828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:17.814776346 +0000 UTC m=+2814.930695952" watchObservedRunningTime="2026-10-29 23:30:17.819775828 +0000 UTC m=+2814.935695433" Oct 29 23:30:17.854017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.853959 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" podStartSLOduration=17.853945603 podStartE2EDuration="17.853945603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:17.851322032 +0000 UTC m=+2814.967241639" watchObservedRunningTime="2026-10-29 23:30:17.853945603 +0000 UTC m=+2814.969865209" Oct 29 23:30:17.901862 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:17.901831 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c938b8_9390_4867_b84b_46248fca232a.slice/crio-9d2af7534eee15c07a7926f976efa3fabf33268897810ed0d42a3feb5d10fafa WatchSource:0}: Error finding container 9d2af7534eee15c07a7926f976efa3fabf33268897810ed0d42a3feb5d10fafa: Status 404 returned error can't find the container with id 9d2af7534eee15c07a7926f976efa3fabf33268897810ed0d42a3feb5d10fafa Oct 29 23:30:17.905224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.905196 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9"] Oct 29 23:30:17.996972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.996949 5736 scope.go:122] "RemoveContainer" containerID="450661a923286541c0a4b75a6641c1e29141ffe100db78f6e5b141508ec89cf9" Oct 29 23:30:17.997375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.997176 5736 scope.go:122] "RemoveContainer" containerID="daa0171f8cdec8525830ca5c9b345522ace850f2b3db0bc8b41eafbe47546a59" Oct 29 23:30:17.997484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.997464 5736 scope.go:122] "RemoveContainer" containerID="4e0f5cb2a117c96335fd1286ad453a1f38f46d0f19a9d5e49b50cda00fe053b6" Oct 29 23:30:17.997621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:17.997603 5736 scope.go:122] "RemoveContainer" containerID="e58de3c7d4e4915369f6d2cd28989d8d7710f535d509123b8f2d2a6c86cdb5fc" Oct 29 23:30:18.794301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.794258 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" event={"ID":"f76f9f99-0971-49bb-b819-23ce4f897001","Type":"ContainerStarted","Data":"4dc519482b99f2fce99ee0532596c764e6f0f2ac3260ff8822c420eee1c21f77"} Oct 29 23:30:18.796328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.796299 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" event={"ID":"ffbe9920-f212-4ec5-8055-754d35f7e76e","Type":"ContainerStarted","Data":"5ea263bac44a96d4b0dc93d25d1541330504fb578a4aa7955743bc9fc8e7ca31"} Oct 29 23:30:18.798306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.798283 5736 generic.go:356] "Generic (PLEG): container finished" podID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerID="5f4a42a2f287eb259b14c72bc800a6f836417d9dc688a1dd0bda28c0db04a8d5" exitCode=60 Oct 29 23:30:18.798422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.798359 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" event={"ID":"08d3c108-ba23-4291-a0cf-909c58d2a8bb","Type":"ContainerDied","Data":"5f4a42a2f287eb259b14c72bc800a6f836417d9dc688a1dd0bda28c0db04a8d5"} Oct 29 23:30:18.798498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.798418 5736 scope.go:122] "RemoveContainer" containerID="2b77db6dae48d3bc751e5cf61da95b470f0f7133998c35bfee54d0904d3c929f" Oct 29 23:30:18.798829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.798814 5736 scope.go:122] "RemoveContainer" containerID="5f4a42a2f287eb259b14c72bc800a6f836417d9dc688a1dd0bda28c0db04a8d5" Oct 29 23:30:18.799210 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:18.799180 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj_odh-ai-gateway-infra(08d3c108-ba23-4291-a0cf-909c58d2a8bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" Oct 29 23:30:18.800544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.800516 5736 generic.go:356] "Generic (PLEG): container finished" podID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerID="0b517256da627703a20b2bc0fc956e507714527d1cd4cba806aed814d1c94d92" exitCode=60 Oct 29 23:30:18.800646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.800563 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" event={"ID":"8d03a3d3-acce-41e7-97ff-47c0beb3a320","Type":"ContainerDied","Data":"0b517256da627703a20b2bc0fc956e507714527d1cd4cba806aed814d1c94d92"} Oct 29 23:30:18.800965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.800947 5736 scope.go:122] "RemoveContainer" containerID="0b517256da627703a20b2bc0fc956e507714527d1cd4cba806aed814d1c94d92" Oct 29 23:30:18.801292 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:18.801266 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv_odh-ai-gateway-infra(8d03a3d3-acce-41e7-97ff-47c0beb3a320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" Oct 29 23:30:18.802694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.802671 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" event={"ID":"c19f4669-9c01-4e64-a396-b755fdbfddd6","Type":"ContainerStarted","Data":"405b99c65ebc8b63b39209ce6c536b292e7d56e0e83ebfac68ccb392571669c6"} Oct 29 23:30:18.804561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.804541 5736 generic.go:356] "Generic (PLEG): container finished" podID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerID="d3d7906c82869d2fb65ca12932ed8c8ed0ff779c7cda23e79235891bf8c2f241" exitCode=60 Oct 29 23:30:18.804677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.804601 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" event={"ID":"46519c4c-5baa-453c-8e06-d2c1e41eb6dd","Type":"ContainerDied","Data":"d3d7906c82869d2fb65ca12932ed8c8ed0ff779c7cda23e79235891bf8c2f241"} Oct 29 23:30:18.804941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.804922 5736 scope.go:122] "RemoveContainer" containerID="d3d7906c82869d2fb65ca12932ed8c8ed0ff779c7cda23e79235891bf8c2f241" Oct 29 23:30:18.805281 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:18.805260 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb_odh-ai-gateway-infra(46519c4c-5baa-453c-8e06-d2c1e41eb6dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" Oct 29 23:30:18.806429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.806404 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" event={"ID":"c6c938b8-9390-4867-b84b-46248fca232a","Type":"ContainerStarted","Data":"6685251cacc33a3e4262a400bb680ae967be03c4af0954cc8ab8e3727ead3669"} Oct 29 23:30:18.806544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.806451 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" event={"ID":"c6c938b8-9390-4867-b84b-46248fca232a","Type":"ContainerStarted","Data":"9d2af7534eee15c07a7926f976efa3fabf33268897810ed0d42a3feb5d10fafa"} Oct 29 23:30:18.806544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.806481 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:18.808569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.808548 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" event={"ID":"ca0cab44-e6f4-44b3-be68-2e14816aeb27","Type":"ContainerStarted","Data":"b16ee34cfe4c5d482ec29fd6b09fca5fe9bbfa17966419539960aeb771313b69"} Oct 29 23:30:18.810533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.810514 5736 generic.go:356] "Generic (PLEG): container finished" podID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerID="272148c246284a76cbfb9e686dca93525e2f40a4827e56e15b3eb10db2012ce8" exitCode=60 Oct 29 23:30:18.810648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.810550 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" event={"ID":"95c61621-cfc4-48fb-ad57-b77f44df64fa","Type":"ContainerDied","Data":"272148c246284a76cbfb9e686dca93525e2f40a4827e56e15b3eb10db2012ce8"} Oct 29 23:30:18.810852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.810838 5736 scope.go:122] "RemoveContainer" containerID="272148c246284a76cbfb9e686dca93525e2f40a4827e56e15b3eb10db2012ce8" Oct 29 23:30:18.811093 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:18.811075 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7_odh-ai-gateway-infra(95c61621-cfc4-48fb-ad57-b77f44df64fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" Oct 29 23:30:18.817727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.817706 5736 scope.go:122] "RemoveContainer" containerID="1c981cef7b9d9780864da4d25ba408e8228be69051b9175e13b5fe3a158994ae" Oct 29 23:30:18.827628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.827600 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb"] Oct 29 23:30:18.836237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.836215 5736 scope.go:122] "RemoveContainer" containerID="8ad5fcd980b629a816766b38648204b0358c80dde9e6ee9763fce525a60e3ccb" Oct 29 23:30:18.852677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.852652 5736 scope.go:122] "RemoveContainer" containerID="241d734b69918dd57ccc90f5d6d7664aabaa25e059db002689c13c6ba0d4a8cb" Oct 29 23:30:18.864338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.864310 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj"] Oct 29 23:30:18.906117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.906042 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7"] Oct 29 23:30:18.941984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.941953 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv"] Oct 29 23:30:18.951064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:18.951016 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" podStartSLOduration=1.951004916 podStartE2EDuration="1.951004916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:18.950348236 +0000 UTC m=+2816.066267853" watchObservedRunningTime="2026-10-29 23:30:18.951004916 +0000 UTC m=+2816.066924523" Oct 29 23:30:19.008257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.008226 5736 scope.go:122] "RemoveContainer" containerID="9f56a276b93a621ae05a97aece529d63e48f10199cfc4f999bd2fa473b670a8d" Oct 29 23:30:19.008635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.008452 5736 scope.go:122] "RemoveContainer" containerID="e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418" Oct 29 23:30:19.008691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.008669 5736 scope.go:122] "RemoveContainer" containerID="027c5750cb0b893676f24e16e7b5db9d69e418c502005a5af7baa8c8e00b0c71" Oct 29 23:30:19.008912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.008874 5736 scope.go:122] "RemoveContainer" containerID="1c0dc01b2a243320e78b5a6643a6824e1e2b243d4212cb9f05e8847a2a7f2806" Oct 29 23:30:19.009564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.009196 5736 scope.go:122] "RemoveContainer" containerID="f0515794027cc0815f0077c7e25f430e1fe6038b22c010e2e246b003b3ba5773" Oct 29 23:30:19.023920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.023877 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" podStartSLOduration=19.023864752 podStartE2EDuration="19.023864752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:19.023241435 +0000 UTC m=+2816.139161041" watchObservedRunningTime="2026-10-29 23:30:19.023864752 +0000 UTC m=+2816.139784368" Oct 29 23:30:19.695865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.695837 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:30:19.821715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.821685 5736 generic.go:356] "Generic (PLEG): container finished" podID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerID="405b99c65ebc8b63b39209ce6c536b292e7d56e0e83ebfac68ccb392571669c6" exitCode=60 Oct 29 23:30:19.821887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.821754 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" event={"ID":"c19f4669-9c01-4e64-a396-b755fdbfddd6","Type":"ContainerDied","Data":"405b99c65ebc8b63b39209ce6c536b292e7d56e0e83ebfac68ccb392571669c6"} Oct 29 23:30:19.821887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.821812 5736 scope.go:122] "RemoveContainer" containerID="4e0f5cb2a117c96335fd1286ad453a1f38f46d0f19a9d5e49b50cda00fe053b6" Oct 29 23:30:19.822337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.822311 5736 scope.go:122] "RemoveContainer" containerID="405b99c65ebc8b63b39209ce6c536b292e7d56e0e83ebfac68ccb392571669c6" Oct 29 23:30:19.822966 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:19.822810 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2_odh-ai-gateway-infra(c19f4669-9c01-4e64-a396-b755fdbfddd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" Oct 29 23:30:19.826052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.826027 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" event={"ID":"470d3dc9-0310-418e-be47-56549cb7573d","Type":"ContainerStarted","Data":"0fe1b90e6a186d3b268d0fa1d300bc26af399aee9e0ca87b581c555b5607c7c0"} Oct 29 23:30:19.828319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.828293 5736 generic.go:356] "Generic (PLEG): container finished" podID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerID="b16ee34cfe4c5d482ec29fd6b09fca5fe9bbfa17966419539960aeb771313b69" exitCode=60 Oct 29 23:30:19.828429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.828317 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" event={"ID":"ca0cab44-e6f4-44b3-be68-2e14816aeb27","Type":"ContainerDied","Data":"b16ee34cfe4c5d482ec29fd6b09fca5fe9bbfa17966419539960aeb771313b69"} Oct 29 23:30:19.828783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.828764 5736 scope.go:122] "RemoveContainer" containerID="b16ee34cfe4c5d482ec29fd6b09fca5fe9bbfa17966419539960aeb771313b69" Oct 29 23:30:19.829170 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:19.829150 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd_odh-ai-gateway-infra(ca0cab44-e6f4-44b3-be68-2e14816aeb27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" Oct 29 23:30:19.834829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.832832 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" event={"ID":"e1e77280-ca64-49f3-a6ca-ee1ec7573142","Type":"ContainerStarted","Data":"6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b"} Oct 29 23:30:19.836218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.836193 5736 generic.go:356] "Generic (PLEG): container finished" podID="f76f9f99-0971-49bb-b819-23ce4f897001" containerID="4dc519482b99f2fce99ee0532596c764e6f0f2ac3260ff8822c420eee1c21f77" exitCode=60 Oct 29 23:30:19.836324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.836287 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" event={"ID":"f76f9f99-0971-49bb-b819-23ce4f897001","Type":"ContainerDied","Data":"4dc519482b99f2fce99ee0532596c764e6f0f2ac3260ff8822c420eee1c21f77"} Oct 29 23:30:19.836727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.836691 5736 scope.go:122] "RemoveContainer" containerID="4dc519482b99f2fce99ee0532596c764e6f0f2ac3260ff8822c420eee1c21f77" Oct 29 23:30:19.837049 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:19.837023 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb_odh-ai-gateway-infra(f76f9f99-0971-49bb-b819-23ce4f897001)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" Oct 29 23:30:19.843737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.843714 5736 scope.go:122] "RemoveContainer" containerID="450661a923286541c0a4b75a6641c1e29141ffe100db78f6e5b141508ec89cf9" Oct 29 23:30:19.864561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.864220 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb"] Oct 29 23:30:19.866936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.866904 5736 generic.go:356] "Generic (PLEG): container finished" podID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerID="5ea263bac44a96d4b0dc93d25d1541330504fb578a4aa7955743bc9fc8e7ca31" exitCode=60 Oct 29 23:30:19.867071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.867007 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" event={"ID":"ffbe9920-f212-4ec5-8055-754d35f7e76e","Type":"ContainerDied","Data":"5ea263bac44a96d4b0dc93d25d1541330504fb578a4aa7955743bc9fc8e7ca31"} Oct 29 23:30:19.867444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.867422 5736 scope.go:122] "RemoveContainer" containerID="5ea263bac44a96d4b0dc93d25d1541330504fb578a4aa7955743bc9fc8e7ca31" Oct 29 23:30:19.867831 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:19.867795 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5_odh-ai-gateway-infra(ffbe9920-f212-4ec5-8055-754d35f7e76e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" Oct 29 23:30:19.896800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.896765 5736 scope.go:122] "RemoveContainer" containerID="daa0171f8cdec8525830ca5c9b345522ace850f2b3db0bc8b41eafbe47546a59" Oct 29 23:30:19.920287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.920247 5736 scope.go:122] "RemoveContainer" containerID="e58de3c7d4e4915369f6d2cd28989d8d7710f535d509123b8f2d2a6c86cdb5fc" Oct 29 23:30:19.924511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.924485 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2"] Oct 29 23:30:19.996693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.996617 5736 scope.go:122] "RemoveContainer" containerID="6aa359e050c3f8183cff0ead51097c3d71192c9360ca544517d6fcb0861ed380" Oct 29 23:30:19.998406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.998383 5736 scope.go:122] "RemoveContainer" containerID="3b8c521790e16ed217b2c00dadc1f3af57f2b9cf6b44a2c09d9b98d1b6de06a5" Oct 29 23:30:19.998977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:19.998955 5736 scope.go:122] "RemoveContainer" containerID="dee99d6fba2b76ea250548deba0af6224bacc2fb6223c9e3dd3240c3d0e25567" Oct 29 23:30:20.044560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.044526 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd"] Oct 29 23:30:20.137520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.137474 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5"] Oct 29 23:30:20.181464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.180250 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" podStartSLOduration=20.180231047 podStartE2EDuration="20.180231047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:20.157144095 +0000 UTC m=+2817.273063701" watchObservedRunningTime="2026-10-29 23:30:20.180231047 +0000 UTC m=+2817.296150654" Oct 29 23:30:20.267268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.267209 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:20.280635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.280354 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-kube-api-access-k42fp\" (UniqueName: \"kubernetes.io/projected/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-kube-api-access-k42fp\") pod \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\" (UID: \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\") " Oct 29 23:30:20.280828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.280570 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-maas-api-service-ca\") pod \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\" (UID: \"46519c4c-5baa-453c-8e06-d2c1e41eb6dd\") " Oct 29 23:30:20.283643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.283233 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-maas-api-service-ca" pod "46519c4c-5baa-453c-8e06-d2c1e41eb6dd" (UID: "46519c4c-5baa-453c-8e06-d2c1e41eb6dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:20.294875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.294847 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-kube-api-access-k42fp" pod "46519c4c-5baa-453c-8e06-d2c1e41eb6dd" (UID: "46519c4c-5baa-453c-8e06-d2c1e41eb6dd"). InnerVolumeSpecName "kube-api-access-k42fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:20.302019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.301992 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:20.310743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.310717 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:20.318567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.318540 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:20.416301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.416259 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.416548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.416530 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k42fp\" (UniqueName: \"kubernetes.io/projected/46519c4c-5baa-453c-8e06-d2c1e41eb6dd-kube-api-access-k42fp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.603651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.536674 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d3c108-ba23-4291-a0cf-909c58d2a8bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d3c108-ba23-4291-a0cf-909c58d2a8bb-maas-api-service-ca\") pod \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\" (UID: \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\") " Oct 29 23:30:20.603651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.536760 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c61621-cfc4-48fb-ad57-b77f44df64fa-kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/95c61621-cfc4-48fb-ad57-b77f44df64fa-kube-api-access-57gkt\") pod \"95c61621-cfc4-48fb-ad57-b77f44df64fa\" (UID: \"95c61621-cfc4-48fb-ad57-b77f44df64fa\") " Oct 29 23:30:20.603651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.536803 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c61621-cfc4-48fb-ad57-b77f44df64fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c61621-cfc4-48fb-ad57-b77f44df64fa-maas-api-service-ca\") pod \"95c61621-cfc4-48fb-ad57-b77f44df64fa\" (UID: \"95c61621-cfc4-48fb-ad57-b77f44df64fa\") " Oct 29 23:30:20.603651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.536885 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d03a3d3-acce-41e7-97ff-47c0beb3a320-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03a3d3-acce-41e7-97ff-47c0beb3a320-maas-api-service-ca\") pod \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\" (UID: \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\") " Oct 29 23:30:20.603651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.536915 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d03a3d3-acce-41e7-97ff-47c0beb3a320-kube-api-access-lhr5w\" (UniqueName: \"kubernetes.io/projected/8d03a3d3-acce-41e7-97ff-47c0beb3a320-kube-api-access-lhr5w\") pod \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\" (UID: \"8d03a3d3-acce-41e7-97ff-47c0beb3a320\") " Oct 29 23:30:20.603651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.537013 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d3c108-ba23-4291-a0cf-909c58d2a8bb-kube-api-access-tmjdz\" (UniqueName: \"kubernetes.io/projected/08d3c108-ba23-4291-a0cf-909c58d2a8bb-kube-api-access-tmjdz\") pod \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\" (UID: \"08d3c108-ba23-4291-a0cf-909c58d2a8bb\") " Oct 29 23:30:20.603651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.565232 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c61621-cfc4-48fb-ad57-b77f44df64fa-maas-api-service-ca" pod "95c61621-cfc4-48fb-ad57-b77f44df64fa" (UID: "95c61621-cfc4-48fb-ad57-b77f44df64fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:20.603651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.567859 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d03a3d3-acce-41e7-97ff-47c0beb3a320-maas-api-service-ca" pod "8d03a3d3-acce-41e7-97ff-47c0beb3a320" (UID: "8d03a3d3-acce-41e7-97ff-47c0beb3a320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:20.603651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.569913 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d3c108-ba23-4291-a0cf-909c58d2a8bb-maas-api-service-ca" pod "08d3c108-ba23-4291-a0cf-909c58d2a8bb" (UID: "08d3c108-ba23-4291-a0cf-909c58d2a8bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:20.637478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.631552 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d03a3d3-acce-41e7-97ff-47c0beb3a320-kube-api-access-lhr5w" pod "8d03a3d3-acce-41e7-97ff-47c0beb3a320" (UID: "8d03a3d3-acce-41e7-97ff-47c0beb3a320"). InnerVolumeSpecName "kube-api-access-lhr5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:20.656129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.639650 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c61621-cfc4-48fb-ad57-b77f44df64fa-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.656129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.639679 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d03a3d3-acce-41e7-97ff-47c0beb3a320-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.656129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.639698 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhr5w\" (UniqueName: \"kubernetes.io/projected/8d03a3d3-acce-41e7-97ff-47c0beb3a320-kube-api-access-lhr5w\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.656129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.639713 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d3c108-ba23-4291-a0cf-909c58d2a8bb-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.700598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.700519 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d3c108-ba23-4291-a0cf-909c58d2a8bb-kube-api-access-tmjdz" pod "08d3c108-ba23-4291-a0cf-909c58d2a8bb" (UID: "08d3c108-ba23-4291-a0cf-909c58d2a8bb"). InnerVolumeSpecName "kube-api-access-tmjdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:20.711294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.711256 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c61621-cfc4-48fb-ad57-b77f44df64fa-kube-api-access-57gkt" pod "95c61621-cfc4-48fb-ad57-b77f44df64fa" (UID: "95c61621-cfc4-48fb-ad57-b77f44df64fa"). InnerVolumeSpecName "kube-api-access-57gkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:20.748153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.741317 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57gkt\" (UniqueName: \"kubernetes.io/projected/95c61621-cfc4-48fb-ad57-b77f44df64fa-kube-api-access-57gkt\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.748153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:20.741353 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmjdz\" (UniqueName: \"kubernetes.io/projected/08d3c108-ba23-4291-a0cf-909c58d2a8bb-kube-api-access-tmjdz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:21.141392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.141361 5736 scope.go:122] "RemoveContainer" containerID="50d9c2f3929decc58a0d456a21517c694dd7f4081d4d2d5ebb032d2ea271802c" Oct 29 23:30:21.146929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.146898 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" event={"ID":"46519c4c-5baa-453c-8e06-d2c1e41eb6dd","Type":"ContainerDied","Data":"5f7dc4aab5e1ebad95b298f892b8e83397cc3ebbdeddb8e9e9d3e6d1830d01d2"} Oct 29 23:30:21.147108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.147093 5736 scope.go:122] "RemoveContainer" containerID="d3d7906c82869d2fb65ca12932ed8c8ed0ff779c7cda23e79235891bf8c2f241" Oct 29 23:30:21.149591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.149556 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb" Oct 29 23:30:21.177167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.177135 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7"] Oct 29 23:30:21.177646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.177617 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" event={"ID":"95c61621-cfc4-48fb-ad57-b77f44df64fa","Type":"ContainerDied","Data":"3f78472645647b22c64774d9c97c828a41afe75de7382db0eee760764856a240"} Oct 29 23:30:21.178524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.178477 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7" Oct 29 23:30:21.183955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.183926 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" event={"ID":"a7da5bb7-45be-4fc5-94d9-de6402481acb","Type":"ContainerStarted","Data":"76492bf84c8ae63e1f8f3ea793b150eeb3f38dc37fbb708aa0648ebc3a0a59c6"} Oct 29 23:30:21.188027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.188002 5736 scope.go:122] "RemoveContainer" containerID="9fb94e26f7937769e1a2066bb39cc836b26ca1497fe20f7c1b310a8a390d52c4" Oct 29 23:30:21.190932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.190792 5736 scope.go:122] "RemoveContainer" containerID="76492bf84c8ae63e1f8f3ea793b150eeb3f38dc37fbb708aa0648ebc3a0a59c6" Oct 29 23:30:21.191350 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:21.191315 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx_odh-ai-gateway-infra(a7da5bb7-45be-4fc5-94d9-de6402481acb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" Oct 29 23:30:21.206655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.204816 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx"] Oct 29 23:30:21.245976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.245766 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" event={"ID":"08d3c108-ba23-4291-a0cf-909c58d2a8bb","Type":"ContainerDied","Data":"e140b35e41b46f6e6b55a68bbcc82530e9b1794751a411aaf1f3d2aeca25cb69"} Oct 29 23:30:21.246470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.246442 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj" Oct 29 23:30:21.254340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.254307 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" event={"ID":"8d03a3d3-acce-41e7-97ff-47c0beb3a320","Type":"ContainerDied","Data":"6af5687abfc3d2643cb149e68d2f2afb112b280048bb538f349490df8bbb8230"} Oct 29 23:30:21.255652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.255626 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv" Oct 29 23:30:21.265413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.265382 5736 generic.go:356] "Generic (PLEG): container finished" podID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerID="beba6881472de44b7281061b83e495506864f918f0be51de023688ae11533d72" exitCode=60 Oct 29 23:30:21.266986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.265632 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" event={"ID":"b8868c3b-f832-4b4c-8968-139096f19a4a","Type":"ContainerDied","Data":"beba6881472de44b7281061b83e495506864f918f0be51de023688ae11533d72"} Oct 29 23:30:21.267694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.267674 5736 scope.go:122] "RemoveContainer" containerID="beba6881472de44b7281061b83e495506864f918f0be51de023688ae11533d72" Oct 29 23:30:21.272135 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:21.271415 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n_odh-ai-gateway-infra(b8868c3b-f832-4b4c-8968-139096f19a4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" Oct 29 23:30:21.291841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.277283 5736 scope.go:122] "RemoveContainer" containerID="272148c246284a76cbfb9e686dca93525e2f40a4827e56e15b3eb10db2012ce8" Oct 29 23:30:21.291841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.283692 5736 generic.go:356] "Generic (PLEG): container finished" podID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerID="0aa7cfeb5ea441145e278d49a249584ee9fa7204f8686269704b900261d651e5" exitCode=60 Oct 29 23:30:21.291841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.283962 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" containerID="cri-o://6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b" gracePeriod=30 Oct 29 23:30:21.291841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.284110 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" event={"ID":"2ab2f333-8e37-4019-ae18-e4a748837c22","Type":"ContainerDied","Data":"0aa7cfeb5ea441145e278d49a249584ee9fa7204f8686269704b900261d651e5"} Oct 29 23:30:21.291841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.284567 5736 scope.go:122] "RemoveContainer" containerID="0aa7cfeb5ea441145e278d49a249584ee9fa7204f8686269704b900261d651e5" Oct 29 23:30:21.291841 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:21.285177 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb_odh-ai-gateway-infra(2ab2f333-8e37-4019-ae18-e4a748837c22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" Oct 29 23:30:21.291841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.285371 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" containerID="cri-o://0fe1b90e6a186d3b268d0fa1d300bc26af399aee9e0ca87b581c555b5607c7c0" gracePeriod=30 Oct 29 23:30:21.404552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.402762 5736 scope.go:122] "RemoveContainer" containerID="5f4a42a2f287eb259b14c72bc800a6f836417d9dc688a1dd0bda28c0db04a8d5" Oct 29 23:30:21.767368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.767322 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv"] Oct 29 23:30:21.899771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.875955 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888610-25pkv"] Oct 29 23:30:21.961809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:21.961749 5736 scope.go:122] "RemoveContainer" containerID="0b517256da627703a20b2bc0fc956e507714527d1cd4cba806aed814d1c94d92" Oct 29 23:30:22.092677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.092617 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb"] Oct 29 23:30:22.120539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.120500 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-45036b-29888610-vjgkb"] Oct 29 23:30:22.128994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.128957 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj"] Oct 29 23:30:22.135792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.135765 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-383f0e-29888610-rfftj"] Oct 29 23:30:22.147494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.147283 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" podUID="93942c3a-1475-4a0d-a9cc-10ec9256b4e6" containerName="payload-processing" containerID="cri-o://8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a" gracePeriod=25 Oct 29 23:30:22.152710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.149720 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7"] Oct 29 23:30:22.152710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.149975 5736 scope.go:122] "RemoveContainer" containerID="1c0dc01b2a243320e78b5a6643a6824e1e2b243d4212cb9f05e8847a2a7f2806" Oct 29 23:30:22.158930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.158739 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888610-668t7"] Oct 29 23:30:22.165442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.161708 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:22.171794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.171774 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:22.294611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.294334 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0cab44-e6f4-44b3-be68-2e14816aeb27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cab44-e6f4-44b3-be68-2e14816aeb27-maas-api-service-ca\") pod \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\" (UID: \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\") " Oct 29 23:30:22.294611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.294421 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c19f4669-9c01-4e64-a396-b755fdbfddd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19f4669-9c01-4e64-a396-b755fdbfddd6-maas-api-service-ca\") pod \"c19f4669-9c01-4e64-a396-b755fdbfddd6\" (UID: \"c19f4669-9c01-4e64-a396-b755fdbfddd6\") " Oct 29 23:30:22.294611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.294555 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0cab44-e6f4-44b3-be68-2e14816aeb27-kube-api-access-drtwc\" (UniqueName: \"kubernetes.io/projected/ca0cab44-e6f4-44b3-be68-2e14816aeb27-kube-api-access-drtwc\") pod \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\" (UID: \"ca0cab44-e6f4-44b3-be68-2e14816aeb27\") " Oct 29 23:30:22.295599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.294828 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19f4669-9c01-4e64-a396-b755fdbfddd6-kube-api-access-4p42v\" (UniqueName: \"kubernetes.io/projected/c19f4669-9c01-4e64-a396-b755fdbfddd6-kube-api-access-4p42v\") pod \"c19f4669-9c01-4e64-a396-b755fdbfddd6\" (UID: \"c19f4669-9c01-4e64-a396-b755fdbfddd6\") " Oct 29 23:30:22.296006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.295984 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0cab44-e6f4-44b3-be68-2e14816aeb27-maas-api-service-ca" pod "ca0cab44-e6f4-44b3-be68-2e14816aeb27" (UID: "ca0cab44-e6f4-44b3-be68-2e14816aeb27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:22.296874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.296303 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19f4669-9c01-4e64-a396-b755fdbfddd6-maas-api-service-ca" pod "c19f4669-9c01-4e64-a396-b755fdbfddd6" (UID: "c19f4669-9c01-4e64-a396-b755fdbfddd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:22.298298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.298269 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19f4669-9c01-4e64-a396-b755fdbfddd6-kube-api-access-4p42v" pod "c19f4669-9c01-4e64-a396-b755fdbfddd6" (UID: "c19f4669-9c01-4e64-a396-b755fdbfddd6"). InnerVolumeSpecName "kube-api-access-4p42v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:22.302404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.302319 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" event={"ID":"ffbe9920-f212-4ec5-8055-754d35f7e76e","Type":"ContainerDied","Data":"4d1e6c5fd1d88f794bc1b80e09d0ec42181cc33bd2454ed4424dfd325c3a409d"} Oct 29 23:30:22.302404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.302343 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0cab44-e6f4-44b3-be68-2e14816aeb27-kube-api-access-drtwc" pod "ca0cab44-e6f4-44b3-be68-2e14816aeb27" (UID: "ca0cab44-e6f4-44b3-be68-2e14816aeb27"). InnerVolumeSpecName "kube-api-access-drtwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:22.302404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.302380 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1e6c5fd1d88f794bc1b80e09d0ec42181cc33bd2454ed4424dfd325c3a409d" Oct 29 23:30:22.319183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.317043 5736 generic.go:356] "Generic (PLEG): container finished" podID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerID="97e6f269dc8e00ab3f9ee706b463f05e890f1e67208e71fcafe363f39fd6547b" exitCode=60 Oct 29 23:30:22.319183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.317175 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" event={"ID":"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2","Type":"ContainerDied","Data":"97e6f269dc8e00ab3f9ee706b463f05e890f1e67208e71fcafe363f39fd6547b"} Oct 29 23:30:22.319183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.317711 5736 scope.go:122] "RemoveContainer" containerID="97e6f269dc8e00ab3f9ee706b463f05e890f1e67208e71fcafe363f39fd6547b" Oct 29 23:30:22.319183 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:22.318138 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw_odh-ai-gateway-infra(21bb330a-9e1a-4f01-ab2f-aabd1c199eb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" Oct 29 23:30:22.324351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.323352 5736 generic.go:356] "Generic (PLEG): container finished" podID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerID="5ec041f7be1dc724884cf2574d15f1ed395dda94c8698a29c01d8e6e8cb8d5bf" exitCode=60 Oct 29 23:30:22.324351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.323479 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" event={"ID":"2b998c5e-6c2a-4065-898e-e4ba61b45a0a","Type":"ContainerDied","Data":"5ec041f7be1dc724884cf2574d15f1ed395dda94c8698a29c01d8e6e8cb8d5bf"} Oct 29 23:30:22.324351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.323535 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx"] Oct 29 23:30:22.324351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.323930 5736 scope.go:122] "RemoveContainer" containerID="5ec041f7be1dc724884cf2574d15f1ed395dda94c8698a29c01d8e6e8cb8d5bf" Oct 29 23:30:22.324351 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:22.324307 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp_odh-ai-gateway-infra(2b998c5e-6c2a-4065-898e-e4ba61b45a0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" Oct 29 23:30:22.332680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.331901 5736 scope.go:122] "RemoveContainer" containerID="027c5750cb0b893676f24e16e7b5db9d69e418c502005a5af7baa8c8e00b0c71" Oct 29 23:30:22.334391 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.334376 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:22.337782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.337085 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" event={"ID":"992c6925-45f5-4cb3-a436-f032da9b30f5","Type":"ContainerStarted","Data":"0c3b83f1a82dcc8659650aaba2dc3c4184f1deb2c52bc2dfe5b67a687160e8ed"} Oct 29 23:30:22.348512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.348338 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:22.348925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.348730 5736 generic.go:356] "Generic (PLEG): container finished" podID="48af7825-6733-4565-8347-58c931a08499" containerID="503e82238a1e3bf737365fb6b52cc48b42a8d9eb35d4bed6eb2d555d7d7cd82f" exitCode=60 Oct 29 23:30:22.348925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.348755 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" event={"ID":"48af7825-6733-4565-8347-58c931a08499","Type":"ContainerDied","Data":"503e82238a1e3bf737365fb6b52cc48b42a8d9eb35d4bed6eb2d555d7d7cd82f"} Oct 29 23:30:22.349305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.349287 5736 scope.go:122] "RemoveContainer" containerID="503e82238a1e3bf737365fb6b52cc48b42a8d9eb35d4bed6eb2d555d7d7cd82f" Oct 29 23:30:22.350093 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:22.349713 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-qfk7d_odh-ai-gateway-infra(48af7825-6733-4565-8347-58c931a08499)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" podUID="48af7825-6733-4565-8347-58c931a08499" Oct 29 23:30:22.350708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.350640 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:22.352667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.351481 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" event={"ID":"c19f4669-9c01-4e64-a396-b755fdbfddd6","Type":"ContainerDied","Data":"ef0bd1e148e0825a9a25db6e185b6f146bf9a2bf94c0f253c192a51915286996"} Oct 29 23:30:22.352667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.351541 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2" Oct 29 23:30:22.357622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.356760 5736 generic.go:356] "Generic (PLEG): container finished" podID="470d3dc9-0310-418e-be47-56549cb7573d" containerID="0fe1b90e6a186d3b268d0fa1d300bc26af399aee9e0ca87b581c555b5607c7c0" exitCode=60 Oct 29 23:30:22.357622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.356836 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" event={"ID":"470d3dc9-0310-418e-be47-56549cb7573d","Type":"ContainerDied","Data":"0fe1b90e6a186d3b268d0fa1d300bc26af399aee9e0ca87b581c555b5607c7c0"} Oct 29 23:30:22.360699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.360680 5736 scope.go:122] "RemoveContainer" containerID="dee99d6fba2b76ea250548deba0af6224bacc2fb6223c9e3dd3240c3d0e25567" Oct 29 23:30:22.360884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.360863 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:22.363359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.363319 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" event={"ID":"ca0cab44-e6f4-44b3-be68-2e14816aeb27","Type":"ContainerDied","Data":"3aec4d584cd76d0ce3271822c3b12a0156dac334c4992d0ec879f9ab43f4a5a5"} Oct 29 23:30:22.363460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.363433 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd" Oct 29 23:30:22.398640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.372706 5736 generic.go:356] "Generic (PLEG): container finished" podID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerID="76492bf84c8ae63e1f8f3ea793b150eeb3f38dc37fbb708aa0648ebc3a0a59c6" exitCode=60 Oct 29 23:30:22.398640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.372858 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" event={"ID":"a7da5bb7-45be-4fc5-94d9-de6402481acb","Type":"ContainerDied","Data":"76492bf84c8ae63e1f8f3ea793b150eeb3f38dc37fbb708aa0648ebc3a0a59c6"} Oct 29 23:30:22.398640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.382884 5736 generic.go:356] "Generic (PLEG): container finished" podID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerID="6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b" exitCode=60 Oct 29 23:30:22.398640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.382974 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" event={"ID":"e1e77280-ca64-49f3-a6ca-ee1ec7573142","Type":"ContainerDied","Data":"6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b"} Oct 29 23:30:22.398640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.383083 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" Oct 29 23:30:22.471376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.410770 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drtwc\" (UniqueName: \"kubernetes.io/projected/ca0cab44-e6f4-44b3-be68-2e14816aeb27-kube-api-access-drtwc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.471376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.410804 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p42v\" (UniqueName: \"kubernetes.io/projected/c19f4669-9c01-4e64-a396-b755fdbfddd6-kube-api-access-4p42v\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.471376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.410819 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cab44-e6f4-44b3-be68-2e14816aeb27-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.471376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.410832 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19f4669-9c01-4e64-a396-b755fdbfddd6-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.512349 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbe9920-f212-4ec5-8055-754d35f7e76e-kube-api-access-7r22q\" (UniqueName: \"kubernetes.io/projected/ffbe9920-f212-4ec5-8055-754d35f7e76e-kube-api-access-7r22q\") pod \"ffbe9920-f212-4ec5-8055-754d35f7e76e\" (UID: \"ffbe9920-f212-4ec5-8055-754d35f7e76e\") " Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.512416 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76f9f99-0971-49bb-b819-23ce4f897001-kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f76f9f99-0971-49bb-b819-23ce4f897001-kube-api-access-2x4lx\") pod \"f76f9f99-0971-49bb-b819-23ce4f897001\" (UID: \"f76f9f99-0971-49bb-b819-23ce4f897001\") " Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.512559 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffbe9920-f212-4ec5-8055-754d35f7e76e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbe9920-f212-4ec5-8055-754d35f7e76e-maas-api-service-ca\") pod \"ffbe9920-f212-4ec5-8055-754d35f7e76e\" (UID: \"ffbe9920-f212-4ec5-8055-754d35f7e76e\") " Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.521114 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470d3dc9-0310-418e-be47-56549cb7573d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/470d3dc9-0310-418e-be47-56549cb7573d-maas-api-service-ca\") pod \"470d3dc9-0310-418e-be47-56549cb7573d\" (UID: \"470d3dc9-0310-418e-be47-56549cb7573d\") " Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.521311 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470d3dc9-0310-418e-be47-56549cb7573d-kube-api-access-sk48p\" (UniqueName: \"kubernetes.io/projected/470d3dc9-0310-418e-be47-56549cb7573d-kube-api-access-sk48p\") pod \"470d3dc9-0310-418e-be47-56549cb7573d\" (UID: \"470d3dc9-0310-418e-be47-56549cb7573d\") " Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.521352 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76f9f99-0971-49bb-b819-23ce4f897001-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76f9f99-0971-49bb-b819-23ce4f897001-maas-api-service-ca\") pod \"f76f9f99-0971-49bb-b819-23ce4f897001\" (UID: \"f76f9f99-0971-49bb-b819-23ce4f897001\") " Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.521463 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e77280-ca64-49f3-a6ca-ee1ec7573142-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e77280-ca64-49f3-a6ca-ee1ec7573142-maas-api-service-ca\") pod \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\" (UID: \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\") " Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.521541 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e77280-ca64-49f3-a6ca-ee1ec7573142-kube-api-access-4g6fw\" (UniqueName: \"kubernetes.io/projected/e1e77280-ca64-49f3-a6ca-ee1ec7573142-kube-api-access-4g6fw\") pod \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\" (UID: \"e1e77280-ca64-49f3-a6ca-ee1ec7573142\") " Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.527329 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470d3dc9-0310-418e-be47-56549cb7573d-maas-api-service-ca" pod "470d3dc9-0310-418e-be47-56549cb7573d" (UID: "470d3dc9-0310-418e-be47-56549cb7573d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.528982 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbe9920-f212-4ec5-8055-754d35f7e76e-maas-api-service-ca" pod "ffbe9920-f212-4ec5-8055-754d35f7e76e" (UID: "ffbe9920-f212-4ec5-8055-754d35f7e76e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.529346 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76f9f99-0971-49bb-b819-23ce4f897001-maas-api-service-ca" pod "f76f9f99-0971-49bb-b819-23ce4f897001" (UID: "f76f9f99-0971-49bb-b819-23ce4f897001"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.530655 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76f9f99-0971-49bb-b819-23ce4f897001-kube-api-access-2x4lx" pod "f76f9f99-0971-49bb-b819-23ce4f897001" (UID: "f76f9f99-0971-49bb-b819-23ce4f897001"). InnerVolumeSpecName "kube-api-access-2x4lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.532806 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e77280-ca64-49f3-a6ca-ee1ec7573142-maas-api-service-ca" pod "e1e77280-ca64-49f3-a6ca-ee1ec7573142" (UID: "e1e77280-ca64-49f3-a6ca-ee1ec7573142"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.533945 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f76f9f99-0971-49bb-b819-23ce4f897001-kube-api-access-2x4lx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.533984 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbe9920-f212-4ec5-8055-754d35f7e76e-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.534002 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/470d3dc9-0310-418e-be47-56549cb7573d-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.534019 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76f9f99-0971-49bb-b819-23ce4f897001-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.534034 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e77280-ca64-49f3-a6ca-ee1ec7573142-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.555097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.546840 5736 scope.go:122] "RemoveContainer" containerID="6aa359e050c3f8183cff0ead51097c3d71192c9360ca544517d6fcb0861ed380" Oct 29 23:30:22.581562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.581508 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e77280-ca64-49f3-a6ca-ee1ec7573142-kube-api-access-4g6fw" pod "e1e77280-ca64-49f3-a6ca-ee1ec7573142" (UID: "e1e77280-ca64-49f3-a6ca-ee1ec7573142"). InnerVolumeSpecName "kube-api-access-4g6fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:22.590863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.590824 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470d3dc9-0310-418e-be47-56549cb7573d-kube-api-access-sk48p" pod "470d3dc9-0310-418e-be47-56549cb7573d" (UID: "470d3dc9-0310-418e-be47-56549cb7573d"). InnerVolumeSpecName "kube-api-access-sk48p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:22.611650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.598656 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbe9920-f212-4ec5-8055-754d35f7e76e-kube-api-access-7r22q" pod "ffbe9920-f212-4ec5-8055-754d35f7e76e" (UID: "ffbe9920-f212-4ec5-8055-754d35f7e76e"). InnerVolumeSpecName "kube-api-access-7r22q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:22.611650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.602920 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2"] Oct 29 23:30:22.613121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.613088 5736 scope.go:122] "RemoveContainer" containerID="3b8c521790e16ed217b2c00dadc1f3af57f2b9cf6b44a2c09d9b98d1b6de06a5" Oct 29 23:30:22.613332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.613235 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n"] Oct 29 23:30:22.625558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.625496 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888610-swbv2"] Oct 29 23:30:22.645831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.637228 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r22q\" (UniqueName: \"kubernetes.io/projected/ffbe9920-f212-4ec5-8055-754d35f7e76e-kube-api-access-7r22q\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.645831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.637259 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk48p\" (UniqueName: \"kubernetes.io/projected/470d3dc9-0310-418e-be47-56549cb7573d-kube-api-access-sk48p\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.645831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.637274 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g6fw\" (UniqueName: \"kubernetes.io/projected/e1e77280-ca64-49f3-a6ca-ee1ec7573142-kube-api-access-4g6fw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:22.654175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.654038 5736 scope.go:122] "RemoveContainer" containerID="405b99c65ebc8b63b39209ce6c536b292e7d56e0e83ebfac68ccb392571669c6" Oct 29 23:30:22.666757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.664378 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f"] Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667095 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667119 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667142 5736 cpu_manager.go:414] "containerMap: removing container" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667150 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667168 5736 cpu_manager.go:414] "containerMap: removing container" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667176 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667187 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667195 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667217 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667224 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667245 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667252 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667262 5736 cpu_manager.go:414] "containerMap: removing container" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667269 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667281 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667289 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667302 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667309 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667321 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667328 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667338 5736 cpu_manager.go:414] "containerMap: removing container" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667345 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667357 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667364 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667379 5736 cpu_manager.go:414] "containerMap: removing container" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667386 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667401 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667410 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667421 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667429 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667447 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667454 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667466 5736 cpu_manager.go:414] "containerMap: removing container" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667474 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667485 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667493 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667508 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667515 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667531 5736 cpu_manager.go:414] "containerMap: removing container" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667538 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667553 5736 cpu_manager.go:414] "containerMap: removing container" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667560 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667598 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667606 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667624 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667631 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667645 5736 cpu_manager.go:414] "containerMap: removing container" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667653 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667665 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667672 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667690 5736 cpu_manager.go:414] "containerMap: removing container" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667698 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667899 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667910 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667920 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667932 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667940 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667950 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667961 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667971 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667980 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.667989 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668002 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668015 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668024 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668033 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668043 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668054 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668064 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668076 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668086 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668094 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerName="cleanup" Oct 29 23:30:22.670503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.668104 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerName="cleanup" Oct 29 23:30:22.696854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.689912 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:22.741937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.711005 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6f162a\"" Oct 29 23:30:22.916632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.806114 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f"] Oct 29 23:30:22.917177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.917082 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:22.917484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.917429 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:22.917749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.917713 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:22.918210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.918168 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwpw\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-kube-api-access-7hwpw\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:22.935080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.927499 5736 scope.go:122] "RemoveContainer" containerID="9f56a276b93a621ae05a97aece529d63e48f10199cfc4f999bd2fa473b670a8d" Oct 29 23:30:23.022805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.991180 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"agc-cred-79a7a2f1e8e5\"" Oct 29 23:30:23.022805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.991309 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"kube-root-ca.crt\"" Oct 29 23:30:23.022805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.991432 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"routing-overlay\"" Oct 29 23:30:23.022805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:22.991511 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"openshift-service-ca.crt\"" Oct 29 23:30:23.097722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.096448 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"payload-processing-external-model-e2e--b1ab4497-dockercfg-zw4x9\"" Oct 29 23:30:23.112464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.111078 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwpw\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-kube-api-access-7hwpw\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:23.112464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.111244 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:23.112464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.111343 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:23.112464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.111435 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:23.250076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.250035 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:23.278397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.278314 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:23.313675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.313638 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwpw\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-kube-api-access-7hwpw\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:23.323244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.323212 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:23.617867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.617666 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6f162a\"" Oct 29 23:30:23.626630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.622436 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:23.641991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.639308 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-routing-overlay-volume\") pod \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " Oct 29 23:30:23.641991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.639691 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-provider-credentials\") pod \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " Oct 29 23:30:23.641991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.639750 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-kube-api-access-zbkt9\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-kube-api-access-zbkt9\") pod \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " Oct 29 23:30:23.641991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.639795 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-plugins-config-volume\") pod \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\" (UID: \"93942c3a-1475-4a0d-a9cc-10ec9256b4e6\") " Oct 29 23:30:23.645055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.644567 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-plugins-config-volume" pod "93942c3a-1475-4a0d-a9cc-10ec9256b4e6" (UID: "93942c3a-1475-4a0d-a9cc-10ec9256b4e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:23.659484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.659412 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-routing-overlay-volume" pod "93942c3a-1475-4a0d-a9cc-10ec9256b4e6" (UID: "93942c3a-1475-4a0d-a9cc-10ec9256b4e6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:23.661912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.661877 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-kube-api-access-zbkt9" pod "93942c3a-1475-4a0d-a9cc-10ec9256b4e6" (UID: "93942c3a-1475-4a0d-a9cc-10ec9256b4e6"). InnerVolumeSpecName "kube-api-access-zbkt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:23.663998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.663610 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-provider-credentials" pod "93942c3a-1475-4a0d-a9cc-10ec9256b4e6" (UID: "93942c3a-1475-4a0d-a9cc-10ec9256b4e6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:23.663998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.663722 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:23.663998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.662535 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f162a\"/\"payload-processing-external-model-e2e--b1ab4497-dockercfg-zw4x9\"" Oct 29 23:30:23.682671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.682635 5736 scope.go:122] "RemoveContainer" containerID="b16ee34cfe4c5d482ec29fd6b09fca5fe9bbfa17966419539960aeb771313b69" Oct 29 23:30:23.820657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.801714 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:23.950693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.950648 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbkt9\" (UniqueName: \"kubernetes.io/projected/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-kube-api-access-zbkt9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:23.950693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.950678 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:23.950852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.950711 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93942c3a-1475-4a0d-a9cc-10ec9256b4e6-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:23.950852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.821201 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:30:23.955602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.955556 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" path="/var/lib/kubelet/pods/08d3c108-ba23-4291-a0cf-909c58d2a8bb/volumes" Oct 29 23:30:23.956394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.956372 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" path="/var/lib/kubelet/pods/46519c4c-5baa-453c-8e06-d2c1e41eb6dd/volumes" Oct 29 23:30:23.957146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.957031 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" path="/var/lib/kubelet/pods/8d03a3d3-acce-41e7-97ff-47c0beb3a320/volumes" Oct 29 23:30:23.964598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.958146 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" path="/var/lib/kubelet/pods/95c61621-cfc4-48fb-ad57-b77f44df64fa/volumes" Oct 29 23:30:23.964598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.958811 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" path="/var/lib/kubelet/pods/c19f4669-9c01-4e64-a396-b755fdbfddd6/volumes" Oct 29 23:30:23.964598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.960351 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb"] Oct 29 23:30:23.964598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.963416 5736 generic.go:356] "Generic (PLEG): container finished" podID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerID="0c3b83f1a82dcc8659650aaba2dc3c4184f1deb2c52bc2dfe5b67a687160e8ed" exitCode=60 Oct 29 23:30:23.964598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.963520 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" event={"ID":"992c6925-45f5-4cb3-a436-f032da9b30f5","Type":"ContainerDied","Data":"0c3b83f1a82dcc8659650aaba2dc3c4184f1deb2c52bc2dfe5b67a687160e8ed"} Oct 29 23:30:23.964598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.963994 5736 scope.go:122] "RemoveContainer" containerID="0c3b83f1a82dcc8659650aaba2dc3c4184f1deb2c52bc2dfe5b67a687160e8ed" Oct 29 23:30:23.964598 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:23.964270 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2_odh-ai-gateway-infra(992c6925-45f5-4cb3-a436-f032da9b30f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" Oct 29 23:30:23.965348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.965331 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:23.967630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.966533 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw"] Oct 29 23:30:23.972928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.972906 5736 generic.go:356] "Generic (PLEG): container finished" podID="93942c3a-1475-4a0d-a9cc-10ec9256b4e6" containerID="8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a" exitCode=0 Oct 29 23:30:23.973128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.973112 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" event={"ID":"93942c3a-1475-4a0d-a9cc-10ec9256b4e6","Type":"ContainerDied","Data":"8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a"} Oct 29 23:30:23.973260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.973246 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" event={"ID":"93942c3a-1475-4a0d-a9cc-10ec9256b4e6","Type":"ContainerDied","Data":"9e9092e770c53a2901dd70d60f87ff1dace52bb8079e1da56fe1f4a42ac54254"} Oct 29 23:30:23.973394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.973381 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4" Oct 29 23:30:23.973936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.973757 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:23.975801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.975775 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" event={"ID":"470d3dc9-0310-418e-be47-56549cb7573d","Type":"ContainerDied","Data":"13d2e6ecf500a21ba6eea8db2b3dcc88b0e9afe4abc52a33bed056bde176a490"} Oct 29 23:30:23.975908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.975850 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7" Oct 29 23:30:23.979847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.979817 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx" event={"ID":"e1e77280-ca64-49f3-a6ca-ee1ec7573142","Type":"ContainerDied","Data":"5b4b015d0b0fc2d087e4726c8a7516923d83b3a53ee826213651218d1b6dbb36"} Oct 29 23:30:23.986858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.986840 5736 scope.go:122] "RemoveContainer" containerID="f0515794027cc0815f0077c7e25f430e1fe6038b22c010e2e246b003b3ba5773" Oct 29 23:30:23.988013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.987998 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" Oct 29 23:30:23.989969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.989942 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2"] Oct 29 23:30:23.990380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.990003 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb" event={"ID":"f76f9f99-0971-49bb-b819-23ce4f897001","Type":"ContainerDied","Data":"78707e26728ace64c418aa7bf2e2dbb0deebf40b6dd6f5ffbba29f0cdcd2cf5b"} Oct 29 23:30:23.990380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.990035 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6"] Oct 29 23:30:23.991294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991274 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerName="cleanup" Oct 29 23:30:23.991365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991300 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerName="cleanup" Oct 29 23:30:23.991365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991362 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerName="cleanup" Oct 29 23:30:23.991473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991371 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerName="cleanup" Oct 29 23:30:23.991473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991381 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" containerName="cleanup" Oct 29 23:30:23.991473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991389 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" containerName="cleanup" Oct 29 23:30:23.991473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991419 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerName="cleanup" Oct 29 23:30:23.991473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991427 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerName="cleanup" Oct 29 23:30:23.991473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991463 5736 cpu_manager.go:414] "containerMap: removing container" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" Oct 29 23:30:23.991473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991471 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991486 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991493 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991512 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991519 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991547 5736 cpu_manager.go:414] "containerMap: removing container" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991554 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c61621-cfc4-48fb-ad57-b77f44df64fa" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991643 5736 cpu_manager.go:414] "containerMap: removing container" podUID="93942c3a-1475-4a0d-a9cc-10ec9256b4e6" containerName="payload-processing" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991653 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93942c3a-1475-4a0d-a9cc-10ec9256b4e6" containerName="payload-processing" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991670 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991677 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991865 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991898 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="470d3dc9-0310-418e-be47-56549cb7573d" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991912 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19f4669-9c01-4e64-a396-b755fdbfddd6" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991924 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d03a3d3-acce-41e7-97ff-47c0beb3a320" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991932 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991948 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991961 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991975 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="93942c3a-1475-4a0d-a9cc-10ec9256b4e6" containerName="payload-processing" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991983 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d3c108-ba23-4291-a0cf-909c58d2a8bb" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.991994 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.992004 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.992013 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.992030 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="46519c4c-5baa-453c-8e06-d2c1e41eb6dd" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.992044 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerName="cleanup" Oct 29 23:30:23.992045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:23.992055 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" containerName="cleanup" Oct 29 23:30:24.001760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.000318 5736 generic.go:356] "Generic (PLEG): container finished" podID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerID="85449d2c93d1a8c75956a68660382bdc00d91779ea4da6b42253de0013d5c625" exitCode=60 Oct 29 23:30:24.011091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.009098 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6"] Oct 29 23:30:24.011091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.009215 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" event={"ID":"8aad335b-fbf2-41ff-b597-ce071f6837e6","Type":"ContainerDied","Data":"85449d2c93d1a8c75956a68660382bdc00d91779ea4da6b42253de0013d5c625"} Oct 29 23:30:24.011091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.009402 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.011091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.009783 5736 scope.go:122] "RemoveContainer" containerID="85449d2c93d1a8c75956a68660382bdc00d91779ea4da6b42253de0013d5c625" Oct 29 23:30:24.011091 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:24.010200 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh_odh-ai-gateway-infra(8aad335b-fbf2-41ff-b597-ce071f6837e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" Oct 29 23:30:24.016619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.016564 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n" Oct 29 23:30:24.019077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.019057 5736 scope.go:122] "RemoveContainer" containerID="6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b" Oct 29 23:30:24.023961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.023913 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"openshift-service-ca.crt\"" Oct 29 23:30:24.023961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.023940 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"kube-root-ca.crt\"" Oct 29 23:30:24.024129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.024095 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fb367d\"" Oct 29 23:30:24.024204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.024172 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"routing-overlay\"" Oct 29 23:30:24.024403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.024382 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9"] Oct 29 23:30:24.028846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.028810 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5" Oct 29 23:30:24.032127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.032099 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"agc-cred-3c593fb49c43\"" Oct 29 23:30:24.032386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.032234 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb367d\"/\"payload-processing-external-model-e2e--06d8c548-dockercfg-8nrlb\"" Oct 29 23:30:24.033566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.033542 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:24.055785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.055755 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7da5bb7-45be-4fc5-94d9-de6402481acb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7da5bb7-45be-4fc5-94d9-de6402481acb-maas-api-service-ca\") pod \"a7da5bb7-45be-4fc5-94d9-de6402481acb\" (UID: \"a7da5bb7-45be-4fc5-94d9-de6402481acb\") " Oct 29 23:30:24.056009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.055995 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7da5bb7-45be-4fc5-94d9-de6402481acb-kube-api-access-pz92v\" (UniqueName: \"kubernetes.io/projected/a7da5bb7-45be-4fc5-94d9-de6402481acb-kube-api-access-pz92v\") pod \"a7da5bb7-45be-4fc5-94d9-de6402481acb\" (UID: \"a7da5bb7-45be-4fc5-94d9-de6402481acb\") " Oct 29 23:30:24.056123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.056112 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8868c3b-f832-4b4c-8968-139096f19a4a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8868c3b-f832-4b4c-8968-139096f19a4a-maas-api-service-ca\") pod \"b8868c3b-f832-4b4c-8968-139096f19a4a\" (UID: \"b8868c3b-f832-4b4c-8968-139096f19a4a\") " Oct 29 23:30:24.056372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.056361 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8868c3b-f832-4b4c-8968-139096f19a4a-kube-api-access-8vwtw\" (UniqueName: \"kubernetes.io/projected/b8868c3b-f832-4b4c-8968-139096f19a4a-kube-api-access-8vwtw\") pod \"b8868c3b-f832-4b4c-8968-139096f19a4a\" (UID: \"b8868c3b-f832-4b4c-8968-139096f19a4a\") " Oct 29 23:30:24.057047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.057006 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.057759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.057738 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvmnr\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-kube-api-access-rvmnr\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.057963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.057951 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.058034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.058025 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.071877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.060792 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8868c3b-f832-4b4c-8968-139096f19a4a-maas-api-service-ca" pod "b8868c3b-f832-4b4c-8968-139096f19a4a" (UID: "b8868c3b-f832-4b4c-8968-139096f19a4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:24.071877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.066198 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7da5bb7-45be-4fc5-94d9-de6402481acb-maas-api-service-ca" pod "a7da5bb7-45be-4fc5-94d9-de6402481acb" (UID: "a7da5bb7-45be-4fc5-94d9-de6402481acb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:24.071877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.068480 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8868c3b-f832-4b4c-8968-139096f19a4a-kube-api-access-8vwtw" pod "b8868c3b-f832-4b4c-8968-139096f19a4a" (UID: "b8868c3b-f832-4b4c-8968-139096f19a4a"). InnerVolumeSpecName "kube-api-access-8vwtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.071877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.071175 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7da5bb7-45be-4fc5-94d9-de6402481acb-kube-api-access-pz92v" pod "a7da5bb7-45be-4fc5-94d9-de6402481acb" (UID: "a7da5bb7-45be-4fc5-94d9-de6402481acb"). InnerVolumeSpecName "kube-api-access-pz92v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.072129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.071848 5736 scope.go:122] "RemoveContainer" containerID="e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418" Oct 29 23:30:24.095637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.095110 5736 scope.go:122] "RemoveContainer" containerID="50d9c2f3929decc58a0d456a21517c694dd7f4081d4d2d5ebb032d2ea271802c" Oct 29 23:30:24.125301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.125277 5736 scope.go:122] "RemoveContainer" containerID="8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a" Oct 29 23:30:24.139660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.139636 5736 scope.go:122] "RemoveContainer" containerID="8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a" Oct 29 23:30:24.139969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.139923 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx"] Oct 29 23:30:24.140071 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:24.140048 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a\": container with ID starting with 8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a not found: ID does not exist" containerID="8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a" Oct 29 23:30:24.140108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.140082 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a"} err="failed to get container status \"8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a\": rpc error: code = NotFound desc = could not find container \"8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a\": container with ID starting with 8352d0df0d9d62daca90a72454d16c3a8321c0673a0a03e56e115d60b841cf4a not found: ID does not exist" Oct 29 23:30:24.140147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.140108 5736 scope.go:122] "RemoveContainer" containerID="0fe1b90e6a186d3b268d0fa1d300bc26af399aee9e0ca87b581c555b5607c7c0" Oct 29 23:30:24.146154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.146115 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888610-7l6rx"] Oct 29 23:30:24.160923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.160888 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.161069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.161011 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvmnr\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-kube-api-access-rvmnr\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.161172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.161153 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.161283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.161198 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.161414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.161394 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7da5bb7-45be-4fc5-94d9-de6402481acb-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.161650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.161623 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz92v\" (UniqueName: \"kubernetes.io/projected/a7da5bb7-45be-4fc5-94d9-de6402481acb-kube-api-access-pz92v\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.161737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.161657 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8868c3b-f832-4b4c-8968-139096f19a4a-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.161737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.161695 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vwtw\" (UniqueName: \"kubernetes.io/projected/b8868c3b-f832-4b4c-8968-139096f19a4a-kube-api-access-8vwtw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.162550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.162462 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.163602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.163523 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.165084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.165060 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.176862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.176839 5736 scope.go:122] "RemoveContainer" containerID="6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b" Oct 29 23:30:24.177450 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:24.177420 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b\": container with ID starting with 6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b not found: ID does not exist" containerID="6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b" Oct 29 23:30:24.177644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.177461 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b"} err="failed to get container status \"6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b\": rpc error: code = NotFound desc = could not find container \"6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b\": container with ID starting with 6317002437e29a203567ed3311618b284f3c6eca0df73d9c0d415a1ce8eeaa0b not found: ID does not exist" Oct 29 23:30:24.177644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.177494 5736 scope.go:122] "RemoveContainer" containerID="e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418" Oct 29 23:30:24.177857 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:24.177832 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418\": container with ID starting with e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418 not found: ID does not exist" containerID="e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418" Oct 29 23:30:24.177902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.177868 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418"} err="failed to get container status \"e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418\": rpc error: code = NotFound desc = could not find container \"e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418\": container with ID starting with e6e1667a749fcd748e5599fdc30d8441c5d3df262d17f465807a7890eb8d0418 not found: ID does not exist" Oct 29 23:30:24.177902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.177885 5736 scope.go:122] "RemoveContainer" containerID="4dc519482b99f2fce99ee0532596c764e6f0f2ac3260ff8822c420eee1c21f77" Oct 29 23:30:24.184964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.184937 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvmnr\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-kube-api-access-rvmnr\") pod \"payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.229158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.229101 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd"] Oct 29 23:30:24.230761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.230734 5736 scope.go:122] "RemoveContainer" containerID="9fb94e26f7937769e1a2066bb39cc836b26ca1497fe20f7c1b310a8a390d52c4" Oct 29 23:30:24.234236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.234211 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:24.235944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.235921 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e57216-29888610-4zntd"] Oct 29 23:30:24.318149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.318124 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:24.326904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.326879 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f"] Oct 29 23:30:24.330381 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:24.330350 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075bdf77_14ec_417e_bf3c_1405e3970272.slice/crio-4713cb4e47ac58caa957736e0f8b5f61d6d0d14773b8d5eb113c599703859096 WatchSource:0}: Error finding container 4713cb4e47ac58caa957736e0f8b5f61d6d0d14773b8d5eb113c599703859096: Status 404 returned error can't find the container with id 4713cb4e47ac58caa957736e0f8b5f61d6d0d14773b8d5eb113c599703859096 Oct 29 23:30:24.355758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.355733 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:24.356058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.356033 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp"] Oct 29 23:30:24.365160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.365134 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-maas-api-service-ca\") pod \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\" (UID: \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\") " Oct 29 23:30:24.366411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.366291 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-maas-api-service-ca" pod "21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" (UID: "21bb330a-9e1a-4f01-ab2f-aabd1c199eb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:24.366644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.366625 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab2f333-8e37-4019-ae18-e4a748837c22-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab2f333-8e37-4019-ae18-e4a748837c22-maas-api-service-ca\") pod \"2ab2f333-8e37-4019-ae18-e4a748837c22\" (UID: \"2ab2f333-8e37-4019-ae18-e4a748837c22\") " Oct 29 23:30:24.372944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.367092 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab2f333-8e37-4019-ae18-e4a748837c22-maas-api-service-ca" pod "2ab2f333-8e37-4019-ae18-e4a748837c22" (UID: "2ab2f333-8e37-4019-ae18-e4a748837c22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:24.373120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.372941 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab2f333-8e37-4019-ae18-e4a748837c22-kube-api-access-nnws7\" (UniqueName: \"kubernetes.io/projected/2ab2f333-8e37-4019-ae18-e4a748837c22-kube-api-access-nnws7\") pod \"2ab2f333-8e37-4019-ae18-e4a748837c22\" (UID: \"2ab2f333-8e37-4019-ae18-e4a748837c22\") " Oct 29 23:30:24.373120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.372988 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-kube-api-access-srr5r\" (UniqueName: \"kubernetes.io/projected/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-kube-api-access-srr5r\") pod \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\" (UID: \"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2\") " Oct 29 23:30:24.374625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.373826 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.374625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.373850 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab2f333-8e37-4019-ae18-e4a748837c22-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.382055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.382007 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab2f333-8e37-4019-ae18-e4a748837c22-kube-api-access-nnws7" pod "2ab2f333-8e37-4019-ae18-e4a748837c22" (UID: "2ab2f333-8e37-4019-ae18-e4a748837c22"). InnerVolumeSpecName "kube-api-access-nnws7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.384376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.384351 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-kube-api-access-srr5r" pod "21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" (UID: "21bb330a-9e1a-4f01-ab2f-aabd1c199eb2"). InnerVolumeSpecName "kube-api-access-srr5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.434633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.434549 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb"] Oct 29 23:30:24.452488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.452042 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888610-wdqrb"] Oct 29 23:30:24.477829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.477793 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnws7\" (UniqueName: \"kubernetes.io/projected/2ab2f333-8e37-4019-ae18-e4a748837c22-kube-api-access-nnws7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.478041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.478021 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srr5r\" (UniqueName: \"kubernetes.io/projected/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2-kube-api-access-srr5r\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.515999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.515959 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7"] Oct 29 23:30:24.520037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.520009 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888610-sl2x7"] Oct 29 23:30:24.593879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.593784 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5"] Oct 29 23:30:24.599492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.599463 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d5e482-29888610-vx2l5"] Oct 29 23:30:24.605714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.605687 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:24.656288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.656257 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d"] Oct 29 23:30:24.682240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.682210 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-kube-api-access-rlljf\" (UniqueName: \"kubernetes.io/projected/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-kube-api-access-rlljf\") pod \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\" (UID: \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\") " Oct 29 23:30:24.682418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.682275 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-maas-api-service-ca\") pod \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\" (UID: \"2b998c5e-6c2a-4065-898e-e4ba61b45a0a\") " Oct 29 23:30:24.683333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.683154 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-maas-api-service-ca" pod "2b998c5e-6c2a-4065-898e-e4ba61b45a0a" (UID: "2b998c5e-6c2a-4065-898e-e4ba61b45a0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:24.683333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.683310 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.686875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.686851 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-kube-api-access-rlljf" pod "2b998c5e-6c2a-4065-898e-e4ba61b45a0a" (UID: "2b998c5e-6c2a-4065-898e-e4ba61b45a0a"). InnerVolumeSpecName "kube-api-access-rlljf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.766407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.766374 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6"] Oct 29 23:30:24.771015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.770988 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4"] Oct 29 23:30:24.777890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.777865 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-77c9d69db9-rrsq4"] Oct 29 23:30:24.779537 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:24.779503 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfdcacc_5638_4252_9100_5a9ebe006df5.slice/crio-c04daa221c818dbcc8f6c8b98251bd7562dfca0c9ade3e6e695dac83c02d2def WatchSource:0}: Error finding container c04daa221c818dbcc8f6c8b98251bd7562dfca0c9ade3e6e695dac83c02d2def: Status 404 returned error can't find the container with id c04daa221c818dbcc8f6c8b98251bd7562dfca0c9ade3e6e695dac83c02d2def Oct 29 23:30:24.785299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.785274 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlljf\" (UniqueName: \"kubernetes.io/projected/2b998c5e-6c2a-4065-898e-e4ba61b45a0a-kube-api-access-rlljf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.819734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.819708 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:24.886900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.886863 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48af7825-6733-4565-8347-58c931a08499-kube-api-access-25smj\" (UniqueName: \"kubernetes.io/projected/48af7825-6733-4565-8347-58c931a08499-kube-api-access-25smj\") pod \"48af7825-6733-4565-8347-58c931a08499\" (UID: \"48af7825-6733-4565-8347-58c931a08499\") " Oct 29 23:30:24.887308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.887279 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48af7825-6733-4565-8347-58c931a08499-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48af7825-6733-4565-8347-58c931a08499-maas-api-service-ca\") pod \"48af7825-6733-4565-8347-58c931a08499\" (UID: \"48af7825-6733-4565-8347-58c931a08499\") " Oct 29 23:30:24.888717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.888685 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48af7825-6733-4565-8347-58c931a08499-maas-api-service-ca" pod "48af7825-6733-4565-8347-58c931a08499" (UID: "48af7825-6733-4565-8347-58c931a08499"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:24.889283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.889259 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48af7825-6733-4565-8347-58c931a08499-kube-api-access-25smj" pod "48af7825-6733-4565-8347-58c931a08499" (UID: "48af7825-6733-4565-8347-58c931a08499"). InnerVolumeSpecName "kube-api-access-25smj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:24.959811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.959775 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n"] Oct 29 23:30:24.965330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.965300 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-a4f2db-29888610-ns54n"] Oct 29 23:30:24.989981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.989948 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25smj\" (UniqueName: \"kubernetes.io/projected/48af7825-6733-4565-8347-58c931a08499-kube-api-access-25smj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:24.989981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:24.989978 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48af7825-6733-4565-8347-58c931a08499-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:25.009334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.009301 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470d3dc9-0310-418e-be47-56549cb7573d" path="/var/lib/kubelet/pods/470d3dc9-0310-418e-be47-56549cb7573d/volumes" Oct 29 23:30:25.010115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.010084 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93942c3a-1475-4a0d-a9cc-10ec9256b4e6" path="/var/lib/kubelet/pods/93942c3a-1475-4a0d-a9cc-10ec9256b4e6/volumes" Oct 29 23:30:25.010667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.010648 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8868c3b-f832-4b4c-8968-139096f19a4a" path="/var/lib/kubelet/pods/b8868c3b-f832-4b4c-8968-139096f19a4a/volumes" Oct 29 23:30:25.011482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.011462 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0cab44-e6f4-44b3-be68-2e14816aeb27" path="/var/lib/kubelet/pods/ca0cab44-e6f4-44b3-be68-2e14816aeb27/volumes" Oct 29 23:30:25.012032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.012010 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e77280-ca64-49f3-a6ca-ee1ec7573142" path="/var/lib/kubelet/pods/e1e77280-ca64-49f3-a6ca-ee1ec7573142/volumes" Oct 29 23:30:25.012454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.012439 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76f9f99-0971-49bb-b819-23ce4f897001" path="/var/lib/kubelet/pods/f76f9f99-0971-49bb-b819-23ce4f897001/volumes" Oct 29 23:30:25.013171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.013152 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffbe9920-f212-4ec5-8055-754d35f7e76e" path="/var/lib/kubelet/pods/ffbe9920-f212-4ec5-8055-754d35f7e76e/volumes" Oct 29 23:30:25.042566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.042538 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" Oct 29 23:30:25.042704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.042555 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp" event={"ID":"2b998c5e-6c2a-4065-898e-e4ba61b45a0a","Type":"ContainerDied","Data":"2e33597b6db65a88943dfc7f9de502867f5974db66b01ce807cd17cd3c8ae52d"} Oct 29 23:30:25.042704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.042661 5736 scope.go:122] "RemoveContainer" containerID="5ec041f7be1dc724884cf2574d15f1ed395dda94c8698a29c01d8e6e8cb8d5bf" Oct 29 23:30:25.044372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.044344 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" Oct 29 23:30:25.044472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.044370 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb" event={"ID":"2ab2f333-8e37-4019-ae18-e4a748837c22","Type":"ContainerDied","Data":"af8fbeeef9da56e7851a64d2e9ea4c7b8e02c36c8b4d826d0f0422fa01c158d0"} Oct 29 23:30:25.048203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.048180 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" event={"ID":"48af7825-6733-4565-8347-58c931a08499","Type":"ContainerDied","Data":"fd5c6aa02ac6edfba46b37f3747ed7d72fad788928d858eb686718703d8c39bf"} Oct 29 23:30:25.048361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.048180 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d" Oct 29 23:30:25.050325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.050301 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" event={"ID":"075bdf77-14ec-417e-bf3c-1405e3970272","Type":"ContainerStarted","Data":"6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460"} Oct 29 23:30:25.050429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.050395 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" event={"ID":"075bdf77-14ec-417e-bf3c-1405e3970272","Type":"ContainerStarted","Data":"4713cb4e47ac58caa957736e0f8b5f61d6d0d14773b8d5eb113c599703859096"} Oct 29 23:30:25.050510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.050472 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:25.055110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.055084 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" event={"ID":"a7da5bb7-45be-4fc5-94d9-de6402481acb","Type":"ContainerDied","Data":"8fc502e154d347778f44fff27e3eaa800168c3f0a57897228e1314a2fb494304"} Oct 29 23:30:25.055110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.055101 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx" Oct 29 23:30:25.056712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.056691 5736 scope.go:122] "RemoveContainer" containerID="0aa7cfeb5ea441145e278d49a249584ee9fa7204f8686269704b900261d651e5" Oct 29 23:30:25.061253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.060662 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" event={"ID":"3cfdcacc-5638-4252-9100-5a9ebe006df5","Type":"ContainerStarted","Data":"97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe"} Oct 29 23:30:25.061253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.060712 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" event={"ID":"3cfdcacc-5638-4252-9100-5a9ebe006df5","Type":"ContainerStarted","Data":"c04daa221c818dbcc8f6c8b98251bd7562dfca0c9ade3e6e695dac83c02d2def"} Oct 29 23:30:25.061253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.060736 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:25.064355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.064331 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" event={"ID":"21bb330a-9e1a-4f01-ab2f-aabd1c199eb2","Type":"ContainerDied","Data":"93fd97653e14a04d6a88b9f9b9674b587180fcf32cc5a973443d7f56da87aa22"} Oct 29 23:30:25.064355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.064351 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw" Oct 29 23:30:25.071357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.071335 5736 scope.go:122] "RemoveContainer" containerID="503e82238a1e3bf737365fb6b52cc48b42a8d9eb35d4bed6eb2d555d7d7cd82f" Oct 29 23:30:25.083625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.083568 5736 scope.go:122] "RemoveContainer" containerID="76492bf84c8ae63e1f8f3ea793b150eeb3f38dc37fbb708aa0648ebc3a0a59c6" Oct 29 23:30:25.131896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.131824 5736 scope.go:122] "RemoveContainer" containerID="97e6f269dc8e00ab3f9ee706b463f05e890f1e67208e71fcafe363f39fd6547b" Oct 29 23:30:25.133789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.132644 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" podStartSLOduration=3.132626404 podStartE2EDuration="3.132626404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:25.130239608 +0000 UTC m=+2822.246159215" watchObservedRunningTime="2026-10-29 23:30:25.132626404 +0000 UTC m=+2822.248546011" Oct 29 23:30:25.162603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.162553 5736 scope.go:122] "RemoveContainer" containerID="beba6881472de44b7281061b83e495506864f918f0be51de023688ae11533d72" Oct 29 23:30:25.171848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.171810 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp"] Oct 29 23:30:25.178590 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.178545 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888610-vpmqp"] Oct 29 23:30:25.205533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.205496 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d"] Oct 29 23:30:25.209816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.209786 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-qfk7d"] Oct 29 23:30:25.278827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.278735 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:25.285404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.285369 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb"] Oct 29 23:30:25.298098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.298059 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888610-v2hqb"] Oct 29 23:30:25.333617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.330888 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" podStartSLOduration=2.330869184 podStartE2EDuration="2.330869184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:25.328289874 +0000 UTC m=+2822.444209480" watchObservedRunningTime="2026-10-29 23:30:25.330869184 +0000 UTC m=+2822.446788794" Oct 29 23:30:25.386793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.386721 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx"] Oct 29 23:30:25.400605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.394862 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888610-tbrzx"] Oct 29 23:30:25.400605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.397708 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992c6925-45f5-4cb3-a436-f032da9b30f5-kube-api-access-njftq\" (UniqueName: \"kubernetes.io/projected/992c6925-45f5-4cb3-a436-f032da9b30f5-kube-api-access-njftq\") pod \"992c6925-45f5-4cb3-a436-f032da9b30f5\" (UID: \"992c6925-45f5-4cb3-a436-f032da9b30f5\") " Oct 29 23:30:25.400605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.397794 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992c6925-45f5-4cb3-a436-f032da9b30f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c6925-45f5-4cb3-a436-f032da9b30f5-maas-api-service-ca\") pod \"992c6925-45f5-4cb3-a436-f032da9b30f5\" (UID: \"992c6925-45f5-4cb3-a436-f032da9b30f5\") " Oct 29 23:30:25.400605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.398117 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992c6925-45f5-4cb3-a436-f032da9b30f5-maas-api-service-ca" pod "992c6925-45f5-4cb3-a436-f032da9b30f5" (UID: "992c6925-45f5-4cb3-a436-f032da9b30f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:25.400605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.398563 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992c6925-45f5-4cb3-a436-f032da9b30f5-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:25.404637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.402007 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992c6925-45f5-4cb3-a436-f032da9b30f5-kube-api-access-njftq" pod "992c6925-45f5-4cb3-a436-f032da9b30f5" (UID: "992c6925-45f5-4cb3-a436-f032da9b30f5"). InnerVolumeSpecName "kube-api-access-njftq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:25.528196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.528114 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw"] Oct 29 23:30:25.534404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.534366 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888610-88rjw"] Oct 29 23:30:25.536421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.536387 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njftq\" (UniqueName: \"kubernetes.io/projected/992c6925-45f5-4cb3-a436-f032da9b30f5-kube-api-access-njftq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:25.645336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.645258 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh"] Oct 29 23:30:25.812939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.812910 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:25.943657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.943541 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aad335b-fbf2-41ff-b597-ce071f6837e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aad335b-fbf2-41ff-b597-ce071f6837e6-maas-api-service-ca\") pod \"8aad335b-fbf2-41ff-b597-ce071f6837e6\" (UID: \"8aad335b-fbf2-41ff-b597-ce071f6837e6\") " Oct 29 23:30:25.943657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.943625 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aad335b-fbf2-41ff-b597-ce071f6837e6-kube-api-access-hk6hh\" (UniqueName: \"kubernetes.io/projected/8aad335b-fbf2-41ff-b597-ce071f6837e6-kube-api-access-hk6hh\") pod \"8aad335b-fbf2-41ff-b597-ce071f6837e6\" (UID: \"8aad335b-fbf2-41ff-b597-ce071f6837e6\") " Oct 29 23:30:25.944040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.944014 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aad335b-fbf2-41ff-b597-ce071f6837e6-maas-api-service-ca" pod "8aad335b-fbf2-41ff-b597-ce071f6837e6" (UID: "8aad335b-fbf2-41ff-b597-ce071f6837e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:25.944323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.944298 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aad335b-fbf2-41ff-b597-ce071f6837e6-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:25.946188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:25.946156 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aad335b-fbf2-41ff-b597-ce071f6837e6-kube-api-access-hk6hh" pod "8aad335b-fbf2-41ff-b597-ce071f6837e6" (UID: "8aad335b-fbf2-41ff-b597-ce071f6837e6"). InnerVolumeSpecName "kube-api-access-hk6hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:26.031097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.030921 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m"] Oct 29 23:30:26.034823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.034796 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c767c33f-zxz5m"] Oct 29 23:30:26.046248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.046220 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk6hh\" (UniqueName: \"kubernetes.io/projected/8aad335b-fbf2-41ff-b597-ce071f6837e6-kube-api-access-hk6hh\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:26.081033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.080997 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" Oct 29 23:30:26.081207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.080997 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh" event={"ID":"8aad335b-fbf2-41ff-b597-ce071f6837e6","Type":"ContainerDied","Data":"ee1e2f0b62497841b3a213a5f321ff0c9ca1b3fd1297090ba362cd3d46611cd2"} Oct 29 23:30:26.081304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.081288 5736 scope.go:122] "RemoveContainer" containerID="85449d2c93d1a8c75956a68660382bdc00d91779ea4da6b42253de0013d5c625" Oct 29 23:30:26.087995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.087969 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" Oct 29 23:30:26.088127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.087991 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2" event={"ID":"992c6925-45f5-4cb3-a436-f032da9b30f5","Type":"ContainerDied","Data":"1b87f73df0e82d766400a31007af9b15aab52adb7b7583cfded25979ec14cb8b"} Oct 29 23:30:26.106359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.106343 5736 scope.go:122] "RemoveContainer" containerID="0c3b83f1a82dcc8659650aaba2dc3c4184f1deb2c52bc2dfe5b67a687160e8ed" Oct 29 23:30:26.145370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.145347 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2"] Oct 29 23:30:26.155271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.155247 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888610-mb2j2"] Oct 29 23:30:26.222916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.222872 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh"] Oct 29 23:30:26.238011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.237975 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888610-gxnsh"] Oct 29 23:30:26.624922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.624838 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k"] Oct 29 23:30:26.625884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625857 5736 cpu_manager.go:414] "containerMap: removing container" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerName="cleanup" Oct 29 23:30:26.625884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625881 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625898 5736 cpu_manager.go:414] "containerMap: removing container" podUID="48af7825-6733-4565-8347-58c931a08499" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625905 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48af7825-6733-4565-8347-58c931a08499" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625938 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625946 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625960 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625967 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625976 5736 cpu_manager.go:414] "containerMap: removing container" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.625985 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626000 5736 cpu_manager.go:414] "containerMap: removing container" podUID="48af7825-6733-4565-8347-58c931a08499" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626007 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48af7825-6733-4565-8347-58c931a08499" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626019 5736 cpu_manager.go:414] "containerMap: removing container" podUID="48af7825-6733-4565-8347-58c931a08499" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626026 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48af7825-6733-4565-8347-58c931a08499" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626055 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626063 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerName="cleanup" Oct 29 23:30:26.626090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626094 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626102 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626115 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626123 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626144 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626152 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626168 5736 cpu_manager.go:414] "containerMap: removing container" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626176 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626187 5736 cpu_manager.go:414] "containerMap: removing container" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626195 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626213 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626219 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626228 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626235 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626248 5736 cpu_manager.go:414] "containerMap: removing container" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626255 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626419 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626432 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626443 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626453 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626461 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626470 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626480 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626492 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626503 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="48af7825-6733-4565-8347-58c931a08499" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626518 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626527 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="48af7825-6733-4565-8347-58c931a08499" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626536 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="48af7825-6733-4565-8347-58c931a08499" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626549 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626560 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626568 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerName="cleanup" Oct 29 23:30:26.626886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.626595 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" containerName="cleanup" Oct 29 23:30:26.634825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.634798 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.663516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.661938 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-90f683-openshift-default-dockercfg-tjqwz\"" Oct 29 23:30:26.667522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.667492 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k"] Oct 29 23:30:26.765793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.765757 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.765974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.765949 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-certs\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.766046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.765995 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-credential-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.766102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.766088 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-podinfo\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.766154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.766112 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istiod-ca-cert\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.766211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.766193 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-data\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.766254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.766234 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66jk\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-kube-api-access-g66jk\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.766308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.766296 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-token\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.766354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.766343 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-envoy\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.871197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.871154 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-certs\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.873176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.873150 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-credential-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.881936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.881872 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-podinfo\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.882672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.882649 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istiod-ca-cert\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.896730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.896702 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-data\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.897310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.897288 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66jk\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-kube-api-access-g66jk\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.897873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.897850 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-token\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.898926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.898904 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-envoy\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.899643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.897168 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-data\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.899820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.881780 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-credential-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.901354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.898787 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-podinfo\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.901503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.883456 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istiod-ca-cert\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.901631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.873054 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-certs\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.901739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.901303 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.901882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.901862 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.908851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.908829 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-envoy\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.943664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.939483 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-token\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.944572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.944510 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66jk\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-kube-api-access-g66jk\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:26.973926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:26.973889 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:27.067617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.066501 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" path="/var/lib/kubelet/pods/21bb330a-9e1a-4f01-ab2f-aabd1c199eb2/volumes" Oct 29 23:30:27.117608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.103678 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" path="/var/lib/kubelet/pods/2ab2f333-8e37-4019-ae18-e4a748837c22/volumes" Oct 29 23:30:27.117608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.104462 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" path="/var/lib/kubelet/pods/2b998c5e-6c2a-4065-898e-e4ba61b45a0a/volumes" Oct 29 23:30:27.149826 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:27.141596 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:30:27.149826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.143024 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48af7825-6733-4565-8347-58c931a08499" path="/var/lib/kubelet/pods/48af7825-6733-4565-8347-58c931a08499/volumes" Oct 29 23:30:27.149826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.143866 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e184156-1c95-4871-9104-7a7a7151f73c" path="/var/lib/kubelet/pods/4e184156-1c95-4871-9104-7a7a7151f73c/volumes" Oct 29 23:30:27.149826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.144494 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" path="/var/lib/kubelet/pods/8aad335b-fbf2-41ff-b597-ce071f6837e6/volumes" Oct 29 23:30:27.152230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.152204 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992c6925-45f5-4cb3-a436-f032da9b30f5" path="/var/lib/kubelet/pods/992c6925-45f5-4cb3-a436-f032da9b30f5/volumes" Oct 29 23:30:27.154227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.154180 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7da5bb7-45be-4fc5-94d9-de6402481acb" path="/var/lib/kubelet/pods/a7da5bb7-45be-4fc5-94d9-de6402481acb/volumes" Oct 29 23:30:27.312158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.312097 5736 scope.go:122] "RemoveContainer" containerID="8775832e1b5315d02cc73fb6571a78c02250ff02d558c625c540fdd8aa8a695f" Oct 29 23:30:27.542519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.542494 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k"] Oct 29 23:30:27.545322 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:27.545290 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d0e8aa_75af_49c5_bc60_043e2d3a16df.slice/crio-d4cb10bd9f6d50992b23db33c7c6ea915a43c05797ddf67e96f0c5624a61ac58 WatchSource:0}: Error finding container d4cb10bd9f6d50992b23db33c7c6ea915a43c05797ddf67e96f0c5624a61ac58: Status 404 returned error can't find the container with id d4cb10bd9f6d50992b23db33c7c6ea915a43c05797ddf67e96f0c5624a61ac58 Oct 29 23:30:27.548383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.548343 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:30:27.548485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.548412 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:30:27.548485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:27.548442 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:30:28.204857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:28.204809 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" event={"ID":"60d0e8aa-75af-49c5-bc60-043e2d3a16df","Type":"ContainerStarted","Data":"0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1"} Oct 29 23:30:28.205381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:28.204886 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" event={"ID":"60d0e8aa-75af-49c5-bc60-043e2d3a16df","Type":"ContainerStarted","Data":"d4cb10bd9f6d50992b23db33c7c6ea915a43c05797ddf67e96f0c5624a61ac58"} Oct 29 23:30:28.251416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:28.251352 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" podStartSLOduration=2.251332962 podStartE2EDuration="2.251332962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:28.247183967 +0000 UTC m=+2825.363103572" watchObservedRunningTime="2026-10-29 23:30:28.251332962 +0000 UTC m=+2825.367252569" Oct 29 23:30:28.976520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:28.976488 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:28.982677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:28.982643 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:29.025744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.025647 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" podUID="c6c938b8-9390-4867-b84b-46248fca232a" containerName="payload-processing" containerID="cri-o://6685251cacc33a3e4262a400bb680ae967be03c4af0954cc8ab8e3727ead3669" gracePeriod=25 Oct 29 23:30:29.220735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.220698 5736 generic.go:356] "Generic (PLEG): container finished" podID="c6c938b8-9390-4867-b84b-46248fca232a" containerID="6685251cacc33a3e4262a400bb680ae967be03c4af0954cc8ab8e3727ead3669" exitCode=0 Oct 29 23:30:29.222531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.222498 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" event={"ID":"c6c938b8-9390-4867-b84b-46248fca232a","Type":"ContainerDied","Data":"6685251cacc33a3e4262a400bb680ae967be03c4af0954cc8ab8e3727ead3669"} Oct 29 23:30:29.222677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.222593 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:29.223843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.223669 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:30:29.329801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.329769 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:29.466823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.466783 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-plugins-config-volume\") pod \"c6c938b8-9390-4867-b84b-46248fca232a\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " Oct 29 23:30:29.466977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.466901 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-routing-overlay-volume\") pod \"c6c938b8-9390-4867-b84b-46248fca232a\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " Oct 29 23:30:29.466977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.466948 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-kube-api-access-hxvm6\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-kube-api-access-hxvm6\") pod \"c6c938b8-9390-4867-b84b-46248fca232a\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " Oct 29 23:30:29.467107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.467090 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-provider-credentials\") pod \"c6c938b8-9390-4867-b84b-46248fca232a\" (UID: \"c6c938b8-9390-4867-b84b-46248fca232a\") " Oct 29 23:30:29.467252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.467227 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-plugins-config-volume" pod "c6c938b8-9390-4867-b84b-46248fca232a" (UID: "c6c938b8-9390-4867-b84b-46248fca232a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:29.467290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.467246 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-routing-overlay-volume" pod "c6c938b8-9390-4867-b84b-46248fca232a" (UID: "c6c938b8-9390-4867-b84b-46248fca232a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:29.467824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.467803 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.467888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.467825 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6c938b8-9390-4867-b84b-46248fca232a-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.469212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.469177 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-provider-credentials" pod "c6c938b8-9390-4867-b84b-46248fca232a" (UID: "c6c938b8-9390-4867-b84b-46248fca232a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:29.469324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.469272 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-kube-api-access-hxvm6" pod "c6c938b8-9390-4867-b84b-46248fca232a" (UID: "c6c938b8-9390-4867-b84b-46248fca232a"). InnerVolumeSpecName "kube-api-access-hxvm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:29.569904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.569833 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxvm6\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-kube-api-access-hxvm6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.569904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:29.569864 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6c938b8-9390-4867-b84b-46248fca232a-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:30.231363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:30.231327 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" event={"ID":"c6c938b8-9390-4867-b84b-46248fca232a","Type":"ContainerDied","Data":"9d2af7534eee15c07a7926f976efa3fabf33268897810ed0d42a3feb5d10fafa"} Oct 29 23:30:30.231788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:30.231395 5736 scope.go:122] "RemoveContainer" containerID="6685251cacc33a3e4262a400bb680ae967be03c4af0954cc8ab8e3727ead3669" Oct 29 23:30:30.231788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:30.231336 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9" Oct 29 23:30:30.313103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:30.313065 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9"] Oct 29 23:30:30.325308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:30.325249 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ddc8dbdbf-r8jd9"] Oct 29 23:30:31.012083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:31.012035 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c938b8-9390-4867-b84b-46248fca232a" path="/var/lib/kubelet/pods/c6c938b8-9390-4867-b84b-46248fca232a/volumes" Oct 29 23:30:32.552016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.551984 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9"] Oct 29 23:30:32.552726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552710 5736 cpu_manager.go:414] "containerMap: removing container" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerName="cleanup" Oct 29 23:30:32.552771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552726 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bb330a-9e1a-4f01-ab2f-aabd1c199eb2" containerName="cleanup" Oct 29 23:30:32.552771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552760 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c6c938b8-9390-4867-b84b-46248fca232a" containerName="payload-processing" Oct 29 23:30:32.552771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552765 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c938b8-9390-4867-b84b-46248fca232a" containerName="payload-processing" Oct 29 23:30:32.552860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552787 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerName="cleanup" Oct 29 23:30:32.552860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552793 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerName="cleanup" Oct 29 23:30:32.552860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552799 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerName="cleanup" Oct 29 23:30:32.552860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552804 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerName="cleanup" Oct 29 23:30:32.552981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552911 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b998c5e-6c2a-4065-898e-e4ba61b45a0a" containerName="cleanup" Oct 29 23:30:32.552981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552920 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c938b8-9390-4867-b84b-46248fca232a" containerName="payload-processing" Oct 29 23:30:32.552981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552930 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab2f333-8e37-4019-ae18-e4a748837c22" containerName="cleanup" Oct 29 23:30:32.552981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.552937 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aad335b-fbf2-41ff-b597-ce071f6837e6" containerName="cleanup" Oct 29 23:30:32.558253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.558232 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:32.575973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.575947 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9"] Oct 29 23:30:32.581924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.581901 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-90f683\"" Oct 29 23:30:32.582014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.581984 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-90f683-dockercfg-hqfb5\"" Oct 29 23:30:32.752600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.720073 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476c2a23-7c51-4704-a62b-09a22abc1f2d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9\" (UID: \"476c2a23-7c51-4704-a62b-09a22abc1f2d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:32.820654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.810787 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqlwd\" (UniqueName: \"kubernetes.io/projected/476c2a23-7c51-4704-a62b-09a22abc1f2d-kube-api-access-jqlwd\") pod \"payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9\" (UID: \"476c2a23-7c51-4704-a62b-09a22abc1f2d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:32.929603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.917621 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476c2a23-7c51-4704-a62b-09a22abc1f2d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9\" (UID: \"476c2a23-7c51-4704-a62b-09a22abc1f2d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:32.929603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.917720 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqlwd\" (UniqueName: \"kubernetes.io/projected/476c2a23-7c51-4704-a62b-09a22abc1f2d-kube-api-access-jqlwd\") pod \"payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9\" (UID: \"476c2a23-7c51-4704-a62b-09a22abc1f2d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:32.929603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.918842 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476c2a23-7c51-4704-a62b-09a22abc1f2d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9\" (UID: \"476c2a23-7c51-4704-a62b-09a22abc1f2d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:32.973615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:32.973382 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqlwd\" (UniqueName: \"kubernetes.io/projected/476c2a23-7c51-4704-a62b-09a22abc1f2d-kube-api-access-jqlwd\") pod \"payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9\" (UID: \"476c2a23-7c51-4704-a62b-09a22abc1f2d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:33.172326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:33.172244 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:33.486055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:33.486023 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9"] Oct 29 23:30:33.488003 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:33.487970 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476c2a23_7c51_4704_a62b_09a22abc1f2d.slice/crio-8a60fe27ce3f220a1ee742fc57609ffdc81716fe32eafcb4224c563b72949dd1 WatchSource:0}: Error finding container 8a60fe27ce3f220a1ee742fc57609ffdc81716fe32eafcb4224c563b72949dd1: Status 404 returned error can't find the container with id 8a60fe27ce3f220a1ee742fc57609ffdc81716fe32eafcb4224c563b72949dd1 Oct 29 23:30:34.269552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:34.269505 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" event={"ID":"476c2a23-7c51-4704-a62b-09a22abc1f2d","Type":"ContainerStarted","Data":"b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb"} Oct 29 23:30:34.269965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:34.269575 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:34.269965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:34.269635 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" event={"ID":"476c2a23-7c51-4704-a62b-09a22abc1f2d","Type":"ContainerStarted","Data":"8a60fe27ce3f220a1ee742fc57609ffdc81716fe32eafcb4224c563b72949dd1"} Oct 29 23:30:34.310376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:34.310323 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" podStartSLOduration=2.310308614 podStartE2EDuration="2.310308614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:34.306641377 +0000 UTC m=+2831.422560986" watchObservedRunningTime="2026-10-29 23:30:34.310308614 +0000 UTC m=+2831.426228222" Oct 29 23:30:36.092623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:36.092588 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:30:36.093014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:36.092667 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:30:45.280445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:45.280417 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:30:54.113513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.113478 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd"] Oct 29 23:30:54.118320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.118296 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.133732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.133708 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:30:54.144141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.144118 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:30:54.148704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.148675 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd"] Oct 29 23:30:54.198507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.198482 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzj8t\" (UniqueName: \"kubernetes.io/projected/5c2680b5-bb27-485d-81b3-74583f9e310a-kube-api-access-lzj8t\") pod \"maas-api-revoke-sub-37617a2f-zgrcd\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.198660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.198625 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37617a2f-zgrcd\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.198834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.198813 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37617a2f-zgrcd\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.300075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.300046 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37617a2f-zgrcd\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.300224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.300184 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37617a2f-zgrcd\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.300304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.300291 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzj8t\" (UniqueName: \"kubernetes.io/projected/5c2680b5-bb27-485d-81b3-74583f9e310a-kube-api-access-lzj8t\") pod \"maas-api-revoke-sub-37617a2f-zgrcd\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.300797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.300775 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37617a2f-zgrcd\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.302689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.302666 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37617a2f-zgrcd\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.324109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.324090 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzj8t\" (UniqueName: \"kubernetes.io/projected/5c2680b5-bb27-485d-81b3-74583f9e310a-kube-api-access-lzj8t\") pod \"maas-api-revoke-sub-37617a2f-zgrcd\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.441109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.439878 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:54.797433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:54.797406 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd"] Oct 29 23:30:54.799624 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:30:54.799590 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2680b5_bb27_485d_81b3_74583f9e310a.slice/crio-2bef2b9d3afbf4265dedeb88c0e305279dd65c50a7db54a7e0339a48e838ffa2 WatchSource:0}: Error finding container 2bef2b9d3afbf4265dedeb88c0e305279dd65c50a7db54a7e0339a48e838ffa2: Status 404 returned error can't find the container with id 2bef2b9d3afbf4265dedeb88c0e305279dd65c50a7db54a7e0339a48e838ffa2 Oct 29 23:30:55.588594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:55.588535 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" event={"ID":"5c2680b5-bb27-485d-81b3-74583f9e310a","Type":"ContainerStarted","Data":"afb3b0062d8473490e9a40bb03238e41e704e7a2fa9de20e5238697f71fadd01"} Oct 29 23:30:55.589070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:55.588657 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" event={"ID":"5c2680b5-bb27-485d-81b3-74583f9e310a","Type":"ContainerStarted","Data":"2bef2b9d3afbf4265dedeb88c0e305279dd65c50a7db54a7e0339a48e838ffa2"} Oct 29 23:30:55.637562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:55.637509 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" podStartSLOduration=1.637493643 podStartE2EDuration="1.637493643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:55.634270997 +0000 UTC m=+2852.750190604" watchObservedRunningTime="2026-10-29 23:30:55.637493643 +0000 UTC m=+2852.753413248" Oct 29 23:30:56.597479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:56.597444 5736 generic.go:356] "Generic (PLEG): container finished" podID="5c2680b5-bb27-485d-81b3-74583f9e310a" containerID="afb3b0062d8473490e9a40bb03238e41e704e7a2fa9de20e5238697f71fadd01" exitCode=0 Oct 29 23:30:56.597979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:56.597534 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" event={"ID":"5c2680b5-bb27-485d-81b3-74583f9e310a","Type":"ContainerDied","Data":"afb3b0062d8473490e9a40bb03238e41e704e7a2fa9de20e5238697f71fadd01"} Oct 29 23:30:56.824979 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:30:56.824943 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:30:57.772675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.772651 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:57.866426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.866330 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2680b5-bb27-485d-81b3-74583f9e310a-kube-api-access-lzj8t\" (UniqueName: \"kubernetes.io/projected/5c2680b5-bb27-485d-81b3-74583f9e310a-kube-api-access-lzj8t\") pod \"5c2680b5-bb27-485d-81b3-74583f9e310a\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " Oct 29 23:30:57.866594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.866435 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-xks-ca\") pod \"5c2680b5-bb27-485d-81b3-74583f9e310a\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " Oct 29 23:30:57.866675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.866650 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-openshift-ca\") pod \"5c2680b5-bb27-485d-81b3-74583f9e310a\" (UID: \"5c2680b5-bb27-485d-81b3-74583f9e310a\") " Oct 29 23:30:57.866988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.866957 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-openshift-ca" pod "5c2680b5-bb27-485d-81b3-74583f9e310a" (UID: "5c2680b5-bb27-485d-81b3-74583f9e310a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:57.867396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.867360 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:57.868631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.868605 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2680b5-bb27-485d-81b3-74583f9e310a-kube-api-access-lzj8t" pod "5c2680b5-bb27-485d-81b3-74583f9e310a" (UID: "5c2680b5-bb27-485d-81b3-74583f9e310a"). InnerVolumeSpecName "kube-api-access-lzj8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:57.868882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.868866 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-xks-ca" pod "5c2680b5-bb27-485d-81b3-74583f9e310a" (UID: "5c2680b5-bb27-485d-81b3-74583f9e310a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:57.968484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.968454 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzj8t\" (UniqueName: \"kubernetes.io/projected/5c2680b5-bb27-485d-81b3-74583f9e310a-kube-api-access-lzj8t\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:57.968484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:57.968483 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2680b5-bb27-485d-81b3-74583f9e310a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:30:58.614745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:58.614713 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" Oct 29 23:30:58.614951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:58.614749 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd" event={"ID":"5c2680b5-bb27-485d-81b3-74583f9e310a","Type":"ContainerDied","Data":"2bef2b9d3afbf4265dedeb88c0e305279dd65c50a7db54a7e0339a48e838ffa2"} Oct 29 23:30:58.614951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:30:58.614804 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bef2b9d3afbf4265dedeb88c0e305279dd65c50a7db54a7e0339a48e838ffa2" Oct 29 23:31:05.438951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.438911 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-96c549c77-ghvh7"] Oct 29 23:31:05.440456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.440426 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5c2680b5-bb27-485d-81b3-74583f9e310a" containerName="revoke-keys" Oct 29 23:31:05.440456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.440452 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2680b5-bb27-485d-81b3-74583f9e310a" containerName="revoke-keys" Oct 29 23:31:05.440696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.440682 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2680b5-bb27-485d-81b3-74583f9e310a" containerName="revoke-keys" Oct 29 23:31:05.445781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.445742 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.461560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.461536 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-96c549c77-ghvh7"] Oct 29 23:31:05.652443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.651523 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-provider-credentials\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.652443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.651899 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-routing-overlay-volume\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.652443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.652090 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cczfs\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-kube-api-access-cczfs\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.652443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.652220 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-plugins-config-volume\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.764701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.757472 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-routing-overlay-volume\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.764701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.757655 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cczfs\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-kube-api-access-cczfs\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.764701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.757773 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-plugins-config-volume\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.764701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.757832 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-provider-credentials\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.764701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.761315 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-routing-overlay-volume\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.764701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.762291 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-plugins-config-volume\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.788605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.787324 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cczfs\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-kube-api-access-cczfs\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:05.821633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:05.808517 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-provider-credentials\") pod \"payload-processing-external-model-96c549c77-ghvh7\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:06.062651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:06.062557 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:06.381665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:06.381637 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-96c549c77-ghvh7"] Oct 29 23:31:06.784241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:06.784186 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" event={"ID":"ac08436f-08fa-4bd3-b31f-db0ed8cfa205","Type":"ContainerStarted","Data":"353803a2a0619ac338a0bbbf47b93162215127daa4407c47610d46d55a2b079b"} Oct 29 23:31:06.784748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:06.784266 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:06.784748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:06.784292 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" event={"ID":"ac08436f-08fa-4bd3-b31f-db0ed8cfa205","Type":"ContainerStarted","Data":"32fb0625eabdebb7c74cd9da06f4743488f2d364f52889777db80a7616fa5b44"} Oct 29 23:31:06.824795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:06.824746 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" podStartSLOduration=1.824732085 podStartE2EDuration="1.824732085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:06.822014803 +0000 UTC m=+2863.937934409" watchObservedRunningTime="2026-10-29 23:31:06.824732085 +0000 UTC m=+2863.940651691" Oct 29 23:31:17.834841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:17.834798 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:31:18.026459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:18.026420 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c9c95994-drxzj"] Oct 29 23:31:23.027447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.027407 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" podUID="ce03d100-f93d-461f-8d8e-c80639f5a2b9" containerName="payload-processing" containerID="cri-o://c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a" gracePeriod=25 Oct 29 23:31:23.312020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.311995 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:31:23.452067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.452035 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-kube-api-access-8mhkt\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-kube-api-access-8mhkt\") pod \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " Oct 29 23:31:23.452230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.452077 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-provider-credentials\") pod \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " Oct 29 23:31:23.452230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.452132 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-plugins-config-volume\") pod \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " Oct 29 23:31:23.452230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.452222 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-routing-overlay-volume\") pod \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\" (UID: \"ce03d100-f93d-461f-8d8e-c80639f5a2b9\") " Oct 29 23:31:23.452651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.452625 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-plugins-config-volume" pod "ce03d100-f93d-461f-8d8e-c80639f5a2b9" (UID: "ce03d100-f93d-461f-8d8e-c80639f5a2b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:23.452763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.452667 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-routing-overlay-volume" pod "ce03d100-f93d-461f-8d8e-c80639f5a2b9" (UID: "ce03d100-f93d-461f-8d8e-c80639f5a2b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:23.453106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.453085 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:23.453163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.453112 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce03d100-f93d-461f-8d8e-c80639f5a2b9-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:23.454349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.454329 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-provider-credentials" pod "ce03d100-f93d-461f-8d8e-c80639f5a2b9" (UID: "ce03d100-f93d-461f-8d8e-c80639f5a2b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:23.454407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.454351 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-kube-api-access-8mhkt" pod "ce03d100-f93d-461f-8d8e-c80639f5a2b9" (UID: "ce03d100-f93d-461f-8d8e-c80639f5a2b9"). InnerVolumeSpecName "kube-api-access-8mhkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:23.554974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.554890 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mhkt\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-kube-api-access-8mhkt\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:23.554974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.554921 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce03d100-f93d-461f-8d8e-c80639f5a2b9-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:23.805619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.805518 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz"] Oct 29 23:31:23.807901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.807864 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ce03d100-f93d-461f-8d8e-c80639f5a2b9" containerName="payload-processing" Oct 29 23:31:23.807901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.807893 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce03d100-f93d-461f-8d8e-c80639f5a2b9" containerName="payload-processing" Oct 29 23:31:23.808268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.808249 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce03d100-f93d-461f-8d8e-c80639f5a2b9" containerName="payload-processing" Oct 29 23:31:23.815419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.815388 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:23.828894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.828865 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5cd0f2\"" Oct 29 23:31:23.829039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.829000 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz"] Oct 29 23:31:23.987924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.978379 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2jh\" (UniqueName: \"kubernetes.io/projected/75b35f2a-9440-48e6-b6a3-52dd58320ed0-kube-api-access-gz2jh\") pod \"payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz\" (UID: \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:23.987924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:23.978440 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b35f2a-9440-48e6-b6a3-52dd58320ed0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz\" (UID: \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:24.091662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.091012 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2jh\" (UniqueName: \"kubernetes.io/projected/75b35f2a-9440-48e6-b6a3-52dd58320ed0-kube-api-access-gz2jh\") pod \"payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz\" (UID: \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:24.091662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.091065 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b35f2a-9440-48e6-b6a3-52dd58320ed0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz\" (UID: \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:24.092231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.092203 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b35f2a-9440-48e6-b6a3-52dd58320ed0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz\" (UID: \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:24.182447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.171270 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2jh\" (UniqueName: \"kubernetes.io/projected/75b35f2a-9440-48e6-b6a3-52dd58320ed0-kube-api-access-gz2jh\") pod \"payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz\" (UID: \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:24.243401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.243353 5736 generic.go:356] "Generic (PLEG): container finished" podID="ce03d100-f93d-461f-8d8e-c80639f5a2b9" containerID="c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a" exitCode=0 Oct 29 23:31:24.243814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.243785 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" event={"ID":"ce03d100-f93d-461f-8d8e-c80639f5a2b9","Type":"ContainerDied","Data":"c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a"} Oct 29 23:31:24.243924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.243853 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" event={"ID":"ce03d100-f93d-461f-8d8e-c80639f5a2b9","Type":"ContainerDied","Data":"196f5a75db80387eb63f55d0b909c7ce1eebe507d6a636beb9d27c296b9ee914"} Oct 29 23:31:24.243924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.243886 5736 scope.go:122] "RemoveContainer" containerID="c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a" Oct 29 23:31:24.244088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.244069 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85c9c95994-drxzj" Oct 29 23:31:24.268771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.267817 5736 scope.go:122] "RemoveContainer" containerID="c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a" Oct 29 23:31:24.268771 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:31:24.268199 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a\": container with ID starting with c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a not found: ID does not exist" containerID="c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a" Oct 29 23:31:24.268771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.268236 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a"} err="failed to get container status \"c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a\": rpc error: code = NotFound desc = could not find container \"c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a\": container with ID starting with c4b58e3299966af081a1705f88fcc3223e17a7567bb419541e835abc2fb5f62a not found: ID does not exist" Oct 29 23:31:24.310433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.310399 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c9c95994-drxzj"] Oct 29 23:31:24.314567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.314537 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85c9c95994-drxzj"] Oct 29 23:31:24.435952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.435865 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:24.747107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:24.747078 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz"] Oct 29 23:31:24.748276 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:31:24.748252 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b35f2a_9440_48e6_b6a3_52dd58320ed0.slice/crio-c72801f4139a18bff7d2e0c66118ae02a863413f587446cc930e63bba16aa940 WatchSource:0}: Error finding container c72801f4139a18bff7d2e0c66118ae02a863413f587446cc930e63bba16aa940: Status 404 returned error can't find the container with id c72801f4139a18bff7d2e0c66118ae02a863413f587446cc930e63bba16aa940 Oct 29 23:31:25.007527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:25.007442 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce03d100-f93d-461f-8d8e-c80639f5a2b9" path="/var/lib/kubelet/pods/ce03d100-f93d-461f-8d8e-c80639f5a2b9/volumes" Oct 29 23:31:25.255709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:25.255673 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" event={"ID":"75b35f2a-9440-48e6-b6a3-52dd58320ed0","Type":"ContainerStarted","Data":"dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48"} Oct 29 23:31:25.256190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:25.255732 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" event={"ID":"75b35f2a-9440-48e6-b6a3-52dd58320ed0","Type":"ContainerStarted","Data":"c72801f4139a18bff7d2e0c66118ae02a863413f587446cc930e63bba16aa940"} Oct 29 23:31:25.256190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:25.255804 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:25.295951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:25.295868 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" podStartSLOduration=2.295854694 podStartE2EDuration="2.295854694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:25.292290817 +0000 UTC m=+2882.408210425" watchObservedRunningTime="2026-10-29 23:31:25.295854694 +0000 UTC m=+2882.411774300" Oct 29 23:31:26.825836 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:31:26.825788 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:31:27.551608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:27.550021 5736 scope.go:122] "RemoveContainer" containerID="84f6a1d5d16ad51fe0ebbb7e587cfe01840bc0287019e8e87eb44ab8415ac79e" Oct 29 23:31:27.582678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:27.582651 5736 scope.go:122] "RemoveContainer" containerID="9542e3f5e7674decbbd5a8552bacd1086d906c50dccf7bc2402154d51236669e" Oct 29 23:31:36.268251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:36.268171 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:31:49.436132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.436096 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z"] Oct 29 23:31:49.442645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.442624 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.462398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.462367 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z"] Oct 29 23:31:49.466670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.466645 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:49.466977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.466951 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:31:49.540185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.540152 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076055e9-sgq2z\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.540348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.540209 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076055e9-sgq2z\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.540348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.540340 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4krh\" (UniqueName: \"kubernetes.io/projected/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-kube-api-access-r4krh\") pod \"maas-api-revoke-sub-076055e9-sgq2z\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.642259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.642224 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076055e9-sgq2z\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.642423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.642285 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076055e9-sgq2z\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.642423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.642373 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4krh\" (UniqueName: \"kubernetes.io/projected/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-kube-api-access-r4krh\") pod \"maas-api-revoke-sub-076055e9-sgq2z\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.643007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.642982 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076055e9-sgq2z\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.644683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.644661 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076055e9-sgq2z\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.664630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.664607 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4krh\" (UniqueName: \"kubernetes.io/projected/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-kube-api-access-r4krh\") pod \"maas-api-revoke-sub-076055e9-sgq2z\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:49.757796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:49.757766 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:50.063853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.063819 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z"] Oct 29 23:31:50.065478 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:31:50.065445 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a621e5_1928_4c40_bd6c_b02ee2f82ba2.slice/crio-599a253f321b1772ab73e597afcd9c9bcea1dc7f56009e2b99f9d4a2d3871bc5 WatchSource:0}: Error finding container 599a253f321b1772ab73e597afcd9c9bcea1dc7f56009e2b99f9d4a2d3871bc5: Status 404 returned error can't find the container with id 599a253f321b1772ab73e597afcd9c9bcea1dc7f56009e2b99f9d4a2d3871bc5 Oct 29 23:31:50.083496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.083469 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l"] Oct 29 23:31:50.089845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.089823 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.107908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.107881 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l"] Oct 29 23:31:50.150590 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.150557 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a70739b7-2689l\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.150767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.150691 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54b9\" (UniqueName: \"kubernetes.io/projected/cc793537-95c8-4dbb-9d45-1f292f51b97a-kube-api-access-c54b9\") pod \"maas-api-revoke-sub-a70739b7-2689l\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.150867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.150810 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a70739b7-2689l\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.252105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.252078 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a70739b7-2689l\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.252236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.252189 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54b9\" (UniqueName: \"kubernetes.io/projected/cc793537-95c8-4dbb-9d45-1f292f51b97a-kube-api-access-c54b9\") pod \"maas-api-revoke-sub-a70739b7-2689l\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.252332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.252318 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a70739b7-2689l\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.252969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.252945 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a70739b7-2689l\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.254420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.254399 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a70739b7-2689l\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.274342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.274316 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54b9\" (UniqueName: \"kubernetes.io/projected/cc793537-95c8-4dbb-9d45-1f292f51b97a-kube-api-access-c54b9\") pod \"maas-api-revoke-sub-a70739b7-2689l\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.418841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.418766 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:50.641561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.641515 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" event={"ID":"77a621e5-1928-4c40-bd6c-b02ee2f82ba2","Type":"ContainerStarted","Data":"fe581187574050876cb1382b73f452c1a0dfb07053e00796fada1dc44b33e476"} Oct 29 23:31:50.641987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.641616 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" event={"ID":"77a621e5-1928-4c40-bd6c-b02ee2f82ba2","Type":"ContainerStarted","Data":"599a253f321b1772ab73e597afcd9c9bcea1dc7f56009e2b99f9d4a2d3871bc5"} Oct 29 23:31:50.680367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.680266 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" podStartSLOduration=1.680247827 podStartE2EDuration="1.680247827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:50.678388791 +0000 UTC m=+2907.794308396" watchObservedRunningTime="2026-10-29 23:31:50.680247827 +0000 UTC m=+2907.796167434" Oct 29 23:31:50.734835 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:50.734381 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l"] Oct 29 23:31:50.738891 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:31:50.738854 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc793537_95c8_4dbb_9d45_1f292f51b97a.slice/crio-206c0b679203220a63fec3e80a63ea1e6f0d6885bbb699682796f948e3609ec5 WatchSource:0}: Error finding container 206c0b679203220a63fec3e80a63ea1e6f0d6885bbb699682796f948e3609ec5: Status 404 returned error can't find the container with id 206c0b679203220a63fec3e80a63ea1e6f0d6885bbb699682796f948e3609ec5 Oct 29 23:31:51.651125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:51.651010 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" event={"ID":"cc793537-95c8-4dbb-9d45-1f292f51b97a","Type":"ContainerStarted","Data":"b27b14d1b548cbea4aebed7c3c13b84723533d44fa01a5132d771ab44d5aa652"} Oct 29 23:31:51.651125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:51.651068 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" event={"ID":"cc793537-95c8-4dbb-9d45-1f292f51b97a","Type":"ContainerStarted","Data":"206c0b679203220a63fec3e80a63ea1e6f0d6885bbb699682796f948e3609ec5"} Oct 29 23:31:51.671063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:51.652835 5736 generic.go:356] "Generic (PLEG): container finished" podID="77a621e5-1928-4c40-bd6c-b02ee2f82ba2" containerID="fe581187574050876cb1382b73f452c1a0dfb07053e00796fada1dc44b33e476" exitCode=0 Oct 29 23:31:51.671063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:51.652923 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" event={"ID":"77a621e5-1928-4c40-bd6c-b02ee2f82ba2","Type":"ContainerDied","Data":"fe581187574050876cb1382b73f452c1a0dfb07053e00796fada1dc44b33e476"} Oct 29 23:31:51.691181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:51.691138 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" podStartSLOduration=1.691126101 podStartE2EDuration="1.691126101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:51.687086182 +0000 UTC m=+2908.803005788" watchObservedRunningTime="2026-10-29 23:31:51.691126101 +0000 UTC m=+2908.807045706" Oct 29 23:31:52.665198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.665152 5736 generic.go:356] "Generic (PLEG): container finished" podID="cc793537-95c8-4dbb-9d45-1f292f51b97a" containerID="b27b14d1b548cbea4aebed7c3c13b84723533d44fa01a5132d771ab44d5aa652" exitCode=0 Oct 29 23:31:52.665726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.665227 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" event={"ID":"cc793537-95c8-4dbb-9d45-1f292f51b97a","Type":"ContainerDied","Data":"b27b14d1b548cbea4aebed7c3c13b84723533d44fa01a5132d771ab44d5aa652"} Oct 29 23:31:52.893164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.893134 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:52.926778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.926716 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-kube-api-access-r4krh\" (UniqueName: \"kubernetes.io/projected/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-kube-api-access-r4krh\") pod \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " Oct 29 23:31:52.926906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.926842 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-xks-ca\") pod \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " Oct 29 23:31:52.926961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.926905 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-openshift-ca\") pod \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\" (UID: \"77a621e5-1928-4c40-bd6c-b02ee2f82ba2\") " Oct 29 23:31:52.927294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.927264 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-openshift-ca" pod "77a621e5-1928-4c40-bd6c-b02ee2f82ba2" (UID: "77a621e5-1928-4c40-bd6c-b02ee2f82ba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:52.927594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.927565 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:52.928859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.928837 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-xks-ca" pod "77a621e5-1928-4c40-bd6c-b02ee2f82ba2" (UID: "77a621e5-1928-4c40-bd6c-b02ee2f82ba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:52.928951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:52.928838 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-kube-api-access-r4krh" pod "77a621e5-1928-4c40-bd6c-b02ee2f82ba2" (UID: "77a621e5-1928-4c40-bd6c-b02ee2f82ba2"). InnerVolumeSpecName "kube-api-access-r4krh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:53.029888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.029857 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4krh\" (UniqueName: \"kubernetes.io/projected/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-kube-api-access-r4krh\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:53.029888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.029889 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a621e5-1928-4c40-bd6c-b02ee2f82ba2-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:53.674986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.674956 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" Oct 29 23:31:53.675385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.674989 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z" event={"ID":"77a621e5-1928-4c40-bd6c-b02ee2f82ba2","Type":"ContainerDied","Data":"599a253f321b1772ab73e597afcd9c9bcea1dc7f56009e2b99f9d4a2d3871bc5"} Oct 29 23:31:53.675385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.675037 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="599a253f321b1772ab73e597afcd9c9bcea1dc7f56009e2b99f9d4a2d3871bc5" Oct 29 23:31:53.824864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.824843 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:53.841564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.841536 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc793537-95c8-4dbb-9d45-1f292f51b97a-kube-api-access-c54b9\" (UniqueName: \"kubernetes.io/projected/cc793537-95c8-4dbb-9d45-1f292f51b97a-kube-api-access-c54b9\") pod \"cc793537-95c8-4dbb-9d45-1f292f51b97a\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " Oct 29 23:31:53.841803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.841784 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-xks-ca\") pod \"cc793537-95c8-4dbb-9d45-1f292f51b97a\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " Oct 29 23:31:53.841889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.841835 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-openshift-ca\") pod \"cc793537-95c8-4dbb-9d45-1f292f51b97a\" (UID: \"cc793537-95c8-4dbb-9d45-1f292f51b97a\") " Oct 29 23:31:53.842252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.842226 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-openshift-ca" pod "cc793537-95c8-4dbb-9d45-1f292f51b97a" (UID: "cc793537-95c8-4dbb-9d45-1f292f51b97a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:53.842397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.842378 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:53.843700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.843673 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc793537-95c8-4dbb-9d45-1f292f51b97a-kube-api-access-c54b9" pod "cc793537-95c8-4dbb-9d45-1f292f51b97a" (UID: "cc793537-95c8-4dbb-9d45-1f292f51b97a"). InnerVolumeSpecName "kube-api-access-c54b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:53.843902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.843882 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-xks-ca" pod "cc793537-95c8-4dbb-9d45-1f292f51b97a" (UID: "cc793537-95c8-4dbb-9d45-1f292f51b97a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:53.944192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.944115 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54b9\" (UniqueName: \"kubernetes.io/projected/cc793537-95c8-4dbb-9d45-1f292f51b97a-kube-api-access-c54b9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:53.944192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:53.944148 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc793537-95c8-4dbb-9d45-1f292f51b97a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:31:54.685109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:54.685077 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" Oct 29 23:31:54.685109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:54.685105 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l" event={"ID":"cc793537-95c8-4dbb-9d45-1f292f51b97a","Type":"ContainerDied","Data":"206c0b679203220a63fec3e80a63ea1e6f0d6885bbb699682796f948e3609ec5"} Oct 29 23:31:54.685551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:31:54.685156 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="206c0b679203220a63fec3e80a63ea1e6f0d6885bbb699682796f948e3609ec5" Oct 29 23:31:56.826958 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:31:56.826914 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:32:00.690291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:00.690249 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-96c549c77-ghvh7"] Oct 29 23:32:05.691194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:05.691155 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" podUID="ac08436f-08fa-4bd3-b31f-db0ed8cfa205" containerName="payload-processing" containerID="cri-o://353803a2a0619ac338a0bbbf47b93162215127daa4407c47610d46d55a2b079b" gracePeriod=25 Oct 29 23:32:05.816003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:05.815971 5736 generic.go:356] "Generic (PLEG): container finished" podID="ac08436f-08fa-4bd3-b31f-db0ed8cfa205" containerID="353803a2a0619ac338a0bbbf47b93162215127daa4407c47610d46d55a2b079b" exitCode=0 Oct 29 23:32:05.816118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:05.816044 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" event={"ID":"ac08436f-08fa-4bd3-b31f-db0ed8cfa205","Type":"ContainerDied","Data":"353803a2a0619ac338a0bbbf47b93162215127daa4407c47610d46d55a2b079b"} Oct 29 23:32:05.987103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:05.987075 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:32:06.131794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.131762 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-routing-overlay-volume\") pod \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " Oct 29 23:32:06.131950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.131855 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-plugins-config-volume\") pod \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " Oct 29 23:32:06.131950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.131945 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-provider-credentials\") pod \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " Oct 29 23:32:06.132034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.131989 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-kube-api-access-cczfs\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-kube-api-access-cczfs\") pod \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\" (UID: \"ac08436f-08fa-4bd3-b31f-db0ed8cfa205\") " Oct 29 23:32:06.132138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.132112 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-routing-overlay-volume" pod "ac08436f-08fa-4bd3-b31f-db0ed8cfa205" (UID: "ac08436f-08fa-4bd3-b31f-db0ed8cfa205"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:06.132308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.132287 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-plugins-config-volume" pod "ac08436f-08fa-4bd3-b31f-db0ed8cfa205" (UID: "ac08436f-08fa-4bd3-b31f-db0ed8cfa205"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:06.132611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.132575 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:06.132740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.132612 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:06.134182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.134159 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-kube-api-access-cczfs" pod "ac08436f-08fa-4bd3-b31f-db0ed8cfa205" (UID: "ac08436f-08fa-4bd3-b31f-db0ed8cfa205"). InnerVolumeSpecName "kube-api-access-cczfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:06.134278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.134206 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-provider-credentials" pod "ac08436f-08fa-4bd3-b31f-db0ed8cfa205" (UID: "ac08436f-08fa-4bd3-b31f-db0ed8cfa205"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:06.233730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.233698 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cczfs\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-kube-api-access-cczfs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:06.233730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.233726 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac08436f-08fa-4bd3-b31f-db0ed8cfa205-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:06.825609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.825539 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" Oct 29 23:32:06.826066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.825602 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-96c549c77-ghvh7" event={"ID":"ac08436f-08fa-4bd3-b31f-db0ed8cfa205","Type":"ContainerDied","Data":"32fb0625eabdebb7c74cd9da06f4743488f2d364f52889777db80a7616fa5b44"} Oct 29 23:32:06.826066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.825699 5736 scope.go:122] "RemoveContainer" containerID="353803a2a0619ac338a0bbbf47b93162215127daa4407c47610d46d55a2b079b" Oct 29 23:32:06.874889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.874860 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-96c549c77-ghvh7"] Oct 29 23:32:06.884850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:06.884828 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-96c549c77-ghvh7"] Oct 29 23:32:07.005464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:07.005424 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac08436f-08fa-4bd3-b31f-db0ed8cfa205" path="/var/lib/kubelet/pods/ac08436f-08fa-4bd3-b31f-db0ed8cfa205/volumes" Oct 29 23:32:08.973562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.973526 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw"] Oct 29 23:32:08.974372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.974350 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ac08436f-08fa-4bd3-b31f-db0ed8cfa205" containerName="payload-processing" Oct 29 23:32:08.974372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.974371 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac08436f-08fa-4bd3-b31f-db0ed8cfa205" containerName="payload-processing" Oct 29 23:32:08.974545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.974384 5736 cpu_manager.go:414] "containerMap: removing container" podUID="77a621e5-1928-4c40-bd6c-b02ee2f82ba2" containerName="revoke-keys" Oct 29 23:32:08.974545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.974391 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a621e5-1928-4c40-bd6c-b02ee2f82ba2" containerName="revoke-keys" Oct 29 23:32:08.974545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.974401 5736 cpu_manager.go:414] "containerMap: removing container" podUID="cc793537-95c8-4dbb-9d45-1f292f51b97a" containerName="revoke-keys" Oct 29 23:32:08.974545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.974406 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc793537-95c8-4dbb-9d45-1f292f51b97a" containerName="revoke-keys" Oct 29 23:32:08.974545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.974517 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac08436f-08fa-4bd3-b31f-db0ed8cfa205" containerName="payload-processing" Oct 29 23:32:08.974545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.974525 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a621e5-1928-4c40-bd6c-b02ee2f82ba2" containerName="revoke-keys" Oct 29 23:32:08.974545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.974534 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc793537-95c8-4dbb-9d45-1f292f51b97a" containerName="revoke-keys" Oct 29 23:32:08.979766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:08.979748 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:09.003625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.003561 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-51462d\"" Oct 29 23:32:09.003760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.003595 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-51462d-dockercfg-hmppt\"" Oct 29 23:32:09.009855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.009826 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw"] Oct 29 23:32:09.259660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.238826 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d640b153-10af-4412-8334-97e3f60cb32f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw\" (UID: \"d640b153-10af-4412-8334-97e3f60cb32f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:09.259660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.239108 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dgrb\" (UniqueName: \"kubernetes.io/projected/d640b153-10af-4412-8334-97e3f60cb32f-kube-api-access-9dgrb\") pod \"payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw\" (UID: \"d640b153-10af-4412-8334-97e3f60cb32f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:09.259660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.239999 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n"] Oct 29 23:32:09.355464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.354017 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dgrb\" (UniqueName: \"kubernetes.io/projected/d640b153-10af-4412-8334-97e3f60cb32f-kube-api-access-9dgrb\") pod \"payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw\" (UID: \"d640b153-10af-4412-8334-97e3f60cb32f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:09.355464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.354266 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d640b153-10af-4412-8334-97e3f60cb32f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw\" (UID: \"d640b153-10af-4412-8334-97e3f60cb32f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:09.355464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.355283 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d640b153-10af-4412-8334-97e3f60cb32f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw\" (UID: \"d640b153-10af-4412-8334-97e3f60cb32f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:09.380614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.369704 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:09.380614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.376653 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n"] Oct 29 23:32:09.405425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.405363 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dgrb\" (UniqueName: \"kubernetes.io/projected/d640b153-10af-4412-8334-97e3f60cb32f-kube-api-access-9dgrb\") pod \"payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw\" (UID: \"d640b153-10af-4412-8334-97e3f60cb32f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:09.456222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.456178 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88078911-4efa-4a76-82d1-70bce5369a7c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n\" (UID: \"88078911-4efa-4a76-82d1-70bce5369a7c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:09.456397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.456379 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r7m8\" (UniqueName: \"kubernetes.io/projected/88078911-4efa-4a76-82d1-70bce5369a7c-kube-api-access-2r7m8\") pod \"payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n\" (UID: \"88078911-4efa-4a76-82d1-70bce5369a7c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:09.557805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.557724 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7m8\" (UniqueName: \"kubernetes.io/projected/88078911-4efa-4a76-82d1-70bce5369a7c-kube-api-access-2r7m8\") pod \"payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n\" (UID: \"88078911-4efa-4a76-82d1-70bce5369a7c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:09.557983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.557895 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88078911-4efa-4a76-82d1-70bce5369a7c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n\" (UID: \"88078911-4efa-4a76-82d1-70bce5369a7c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:09.558655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.558632 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88078911-4efa-4a76-82d1-70bce5369a7c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n\" (UID: \"88078911-4efa-4a76-82d1-70bce5369a7c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:09.579424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.579400 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7m8\" (UniqueName: \"kubernetes.io/projected/88078911-4efa-4a76-82d1-70bce5369a7c-kube-api-access-2r7m8\") pod \"payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n\" (UID: \"88078911-4efa-4a76-82d1-70bce5369a7c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:09.594216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.594189 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:09.746653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.746620 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:09.911895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:09.911134 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw"] Oct 29 23:32:09.911958 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:32:09.911896 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd640b153_10af_4412_8334_97e3f60cb32f.slice/crio-3be985af721fd7ca00587047b398d92613ef61fd46b3ebd50b5071484b3ce76e WatchSource:0}: Error finding container 3be985af721fd7ca00587047b398d92613ef61fd46b3ebd50b5071484b3ce76e: Status 404 returned error can't find the container with id 3be985af721fd7ca00587047b398d92613ef61fd46b3ebd50b5071484b3ce76e Oct 29 23:32:10.054092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:10.054057 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n"] Oct 29 23:32:10.056086 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:32:10.056037 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88078911_4efa_4a76_82d1_70bce5369a7c.slice/crio-2c84e64965f84a82bfc07aac58589ed27a6cd2a046908f7e887d6b19a53b7c4c WatchSource:0}: Error finding container 2c84e64965f84a82bfc07aac58589ed27a6cd2a046908f7e887d6b19a53b7c4c: Status 404 returned error can't find the container with id 2c84e64965f84a82bfc07aac58589ed27a6cd2a046908f7e887d6b19a53b7c4c Oct 29 23:32:10.880070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:10.880031 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" event={"ID":"d640b153-10af-4412-8334-97e3f60cb32f","Type":"ContainerStarted","Data":"afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c"} Oct 29 23:32:10.880252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:10.880096 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" event={"ID":"d640b153-10af-4412-8334-97e3f60cb32f","Type":"ContainerStarted","Data":"3be985af721fd7ca00587047b398d92613ef61fd46b3ebd50b5071484b3ce76e"} Oct 29 23:32:10.880252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:10.880124 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:10.881899 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:10.881872 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" event={"ID":"88078911-4efa-4a76-82d1-70bce5369a7c","Type":"ContainerStarted","Data":"adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226"} Oct 29 23:32:10.882038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:10.881920 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" event={"ID":"88078911-4efa-4a76-82d1-70bce5369a7c","Type":"ContainerStarted","Data":"2c84e64965f84a82bfc07aac58589ed27a6cd2a046908f7e887d6b19a53b7c4c"} Oct 29 23:32:10.882038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:10.881961 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:10.919542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:10.919492 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" podStartSLOduration=2.919481181 podStartE2EDuration="2.919481181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:10.916417712 +0000 UTC m=+2928.032337317" watchObservedRunningTime="2026-10-29 23:32:10.919481181 +0000 UTC m=+2928.035400786" Oct 29 23:32:10.954442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:10.954376 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" podStartSLOduration=1.95436347 podStartE2EDuration="1.95436347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:10.95349642 +0000 UTC m=+2928.069416026" watchObservedRunningTime="2026-10-29 23:32:10.95436347 +0000 UTC m=+2928.070283076" Oct 29 23:32:12.002896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:11.999237 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l"] Oct 29 23:32:12.008022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.007997 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.018703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.018655 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l"] Oct 29 23:32:12.109130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.109089 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74krc\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-kube-api-access-74krc\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.109376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.109288 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.109562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.109535 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.109633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.109593 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.219605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.213969 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74krc\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-kube-api-access-74krc\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.219605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.214126 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.219605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.214334 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.219605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.214370 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.219605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.216292 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.231606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.227568 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.231606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.228602 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.257665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.256626 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74krc\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-kube-api-access-74krc\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.335607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.331656 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.622968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.622894 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v"] Oct 29 23:32:12.628716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.628697 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.642448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.642421 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:32:12.642573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.642504 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:32:12.646757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.646717 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v"] Oct 29 23:32:12.747714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.747685 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l"] Oct 29 23:32:12.749552 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:32:12.749525 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf362d65_c122_473d_9359_64d294b3a535.slice/crio-adc02560958397431bcfb94fee8762cb7c6e1d3e4db33a300269958331de6e99 WatchSource:0}: Error finding container adc02560958397431bcfb94fee8762cb7c6e1d3e4db33a300269958331de6e99: Status 404 returned error can't find the container with id adc02560958397431bcfb94fee8762cb7c6e1d3e4db33a300269958331de6e99 Oct 29 23:32:12.769752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.769728 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6211d41d-vjc4v\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.769848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.769781 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6211d41d-vjc4v\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.769929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.769904 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2vn\" (UniqueName: \"kubernetes.io/projected/70c2012a-8e35-439d-bf4b-a610ea623c78-kube-api-access-vl2vn\") pod \"maas-api-revoke-sub-6211d41d-vjc4v\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.871226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.871192 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6211d41d-vjc4v\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.871426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.871259 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6211d41d-vjc4v\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.871426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.871306 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2vn\" (UniqueName: \"kubernetes.io/projected/70c2012a-8e35-439d-bf4b-a610ea623c78-kube-api-access-vl2vn\") pod \"maas-api-revoke-sub-6211d41d-vjc4v\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.871924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.871899 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6211d41d-vjc4v\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.873454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.873411 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6211d41d-vjc4v\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.892272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.892249 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2vn\" (UniqueName: \"kubernetes.io/projected/70c2012a-8e35-439d-bf4b-a610ea623c78-kube-api-access-vl2vn\") pod \"maas-api-revoke-sub-6211d41d-vjc4v\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.922480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.922452 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" event={"ID":"bf362d65-c122-473d-9359-64d294b3a535","Type":"ContainerStarted","Data":"0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1"} Oct 29 23:32:12.922599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.922504 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" event={"ID":"bf362d65-c122-473d-9359-64d294b3a535","Type":"ContainerStarted","Data":"adc02560958397431bcfb94fee8762cb7c6e1d3e4db33a300269958331de6e99"} Oct 29 23:32:12.922599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.922551 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:12.948312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.948290 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:12.963388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:12.963348 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" podStartSLOduration=1.963336795 podStartE2EDuration="1.963336795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:12.959726257 +0000 UTC m=+2930.075645863" watchObservedRunningTime="2026-10-29 23:32:12.963336795 +0000 UTC m=+2930.079256446" Oct 29 23:32:13.262169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:13.262133 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v"] Oct 29 23:32:13.262749 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:32:13.262720 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c2012a_8e35_439d_bf4b_a610ea623c78.slice/crio-4a5264d315eed9c11a6d9734073b78d88bc8acaebff68619cb4119621886e787 WatchSource:0}: Error finding container 4a5264d315eed9c11a6d9734073b78d88bc8acaebff68619cb4119621886e787: Status 404 returned error can't find the container with id 4a5264d315eed9c11a6d9734073b78d88bc8acaebff68619cb4119621886e787 Oct 29 23:32:13.932647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:13.932548 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" event={"ID":"70c2012a-8e35-439d-bf4b-a610ea623c78","Type":"ContainerStarted","Data":"47f12046a58bcc082ab599a172cf85f257c7d999cb240d5d0680785368e50f39"} Oct 29 23:32:13.932647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:13.932629 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" event={"ID":"70c2012a-8e35-439d-bf4b-a610ea623c78","Type":"ContainerStarted","Data":"4a5264d315eed9c11a6d9734073b78d88bc8acaebff68619cb4119621886e787"} Oct 29 23:32:13.971273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:13.971219 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" podStartSLOduration=1.971201968 podStartE2EDuration="1.971201968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:13.968599848 +0000 UTC m=+2931.084519446" watchObservedRunningTime="2026-10-29 23:32:13.971201968 +0000 UTC m=+2931.087121575" Oct 29 23:32:14.942010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:14.941974 5736 generic.go:356] "Generic (PLEG): container finished" podID="70c2012a-8e35-439d-bf4b-a610ea623c78" containerID="47f12046a58bcc082ab599a172cf85f257c7d999cb240d5d0680785368e50f39" exitCode=0 Oct 29 23:32:14.942434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:14.942051 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" event={"ID":"70c2012a-8e35-439d-bf4b-a610ea623c78","Type":"ContainerDied","Data":"47f12046a58bcc082ab599a172cf85f257c7d999cb240d5d0680785368e50f39"} Oct 29 23:32:16.135144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.135116 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:16.234831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.234800 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c2012a-8e35-439d-bf4b-a610ea623c78-kube-api-access-vl2vn\" (UniqueName: \"kubernetes.io/projected/70c2012a-8e35-439d-bf4b-a610ea623c78-kube-api-access-vl2vn\") pod \"70c2012a-8e35-439d-bf4b-a610ea623c78\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " Oct 29 23:32:16.235000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.234847 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-xks-ca\") pod \"70c2012a-8e35-439d-bf4b-a610ea623c78\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " Oct 29 23:32:16.235000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.234976 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-openshift-ca\") pod \"70c2012a-8e35-439d-bf4b-a610ea623c78\" (UID: \"70c2012a-8e35-439d-bf4b-a610ea623c78\") " Oct 29 23:32:16.235322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.235300 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-openshift-ca" pod "70c2012a-8e35-439d-bf4b-a610ea623c78" (UID: "70c2012a-8e35-439d-bf4b-a610ea623c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:16.235787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.235767 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.237016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.236993 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-xks-ca" pod "70c2012a-8e35-439d-bf4b-a610ea623c78" (UID: "70c2012a-8e35-439d-bf4b-a610ea623c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:16.237121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.237100 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c2012a-8e35-439d-bf4b-a610ea623c78-kube-api-access-vl2vn" pod "70c2012a-8e35-439d-bf4b-a610ea623c78" (UID: "70c2012a-8e35-439d-bf4b-a610ea623c78"). InnerVolumeSpecName "kube-api-access-vl2vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:16.337669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.337642 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2vn\" (UniqueName: \"kubernetes.io/projected/70c2012a-8e35-439d-bf4b-a610ea623c78-kube-api-access-vl2vn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.337669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.337669 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c2012a-8e35-439d-bf4b-a610ea623c78-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.960862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.960829 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" Oct 29 23:32:16.961017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.960833 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v" event={"ID":"70c2012a-8e35-439d-bf4b-a610ea623c78","Type":"ContainerDied","Data":"4a5264d315eed9c11a6d9734073b78d88bc8acaebff68619cb4119621886e787"} Oct 29 23:32:16.961017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:16.960959 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a5264d315eed9c11a6d9734073b78d88bc8acaebff68619cb4119621886e787" Oct 29 23:32:21.970601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:21.968364 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:32:21.971362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:21.970715 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:32:23.961855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:23.961819 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:24.105641 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:24.102475 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e2ac7/agc-cred-366eb7011dee: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:24.105641 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:24.102509 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:24.105641 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:24.102604 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials podName:bf362d65-c122-473d-9359-64d294b3a535 nodeName:}" failed. No retries permitted until 2026-10-29 23:32:24.60256286 +0000 UTC m=+2941.718482478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" (UID: "bf362d65-c122-473d-9359-64d294b3a535") : secret "agc-cred-366eb7011dee" not found Oct 29 23:32:24.129969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.129781 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz"] Oct 29 23:32:24.176022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.175797 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr"] Oct 29 23:32:24.177042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.177015 5736 cpu_manager.go:414] "containerMap: removing container" podUID="70c2012a-8e35-439d-bf4b-a610ea623c78" containerName="revoke-keys" Oct 29 23:32:24.177042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.177037 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c2012a-8e35-439d-bf4b-a610ea623c78" containerName="revoke-keys" Oct 29 23:32:24.177346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.177327 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c2012a-8e35-439d-bf4b-a610ea623c78" containerName="revoke-keys" Oct 29 23:32:24.183178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.182304 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.184965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.184916 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr"] Oct 29 23:32:24.294297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.294208 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.294297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.294260 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.294534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.294403 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bd5\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-kube-api-access-w4bd5\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.294648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.294630 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.400604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.400003 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.400604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.400065 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.400604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.400135 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bd5\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-kube-api-access-w4bd5\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.400604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.400272 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.411830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.401475 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.431601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.429219 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.431601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.429947 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.455602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.454130 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bd5\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-kube-api-access-w4bd5\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.524615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.523870 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:24.605055 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:24.605015 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e2ac7/agc-cred-366eb7011dee: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:24.605055 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:24.605058 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:24.605339 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:24.605143 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials podName:bf362d65-c122-473d-9359-64d294b3a535 nodeName:}" failed. No retries permitted until 2026-10-29 23:32:25.605123399 +0000 UTC m=+2942.721043002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" (UID: "bf362d65-c122-473d-9359-64d294b3a535") : secret "agc-cred-366eb7011dee" not found Oct 29 23:32:24.712613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.712520 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl"] Oct 29 23:32:24.717821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.717797 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:24.731663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.731639 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:32:24.732969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.731876 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:32:24.737282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.737235 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl"] Oct 29 23:32:24.810121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.810090 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p95rq\" (UniqueName: \"kubernetes.io/projected/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-kube-api-access-p95rq\") pod \"maas-api-revoke-sub-93c76757-lkvfl\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:24.810305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.810278 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93c76757-lkvfl\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:24.810444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.810428 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93c76757-lkvfl\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:24.882522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.882498 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr"] Oct 29 23:32:24.883882 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:32:24.883853 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ba56da_970e_42e4_9319_c053cd9e76b9.slice/crio-b73b4f0d5ba5e93f14e3951e9ab8a785894aad9ddc076a90b7be3465a56dc784 WatchSource:0}: Error finding container b73b4f0d5ba5e93f14e3951e9ab8a785894aad9ddc076a90b7be3465a56dc784: Status 404 returned error can't find the container with id b73b4f0d5ba5e93f14e3951e9ab8a785894aad9ddc076a90b7be3465a56dc784 Oct 29 23:32:24.911782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.911761 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93c76757-lkvfl\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:24.911910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.911896 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p95rq\" (UniqueName: \"kubernetes.io/projected/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-kube-api-access-p95rq\") pod \"maas-api-revoke-sub-93c76757-lkvfl\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:24.911970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.911957 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93c76757-lkvfl\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:24.912346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.912323 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93c76757-lkvfl\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:24.914132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.914115 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93c76757-lkvfl\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:24.933456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:24.933433 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p95rq\" (UniqueName: \"kubernetes.io/projected/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-kube-api-access-p95rq\") pod \"maas-api-revoke-sub-93c76757-lkvfl\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:25.035739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:25.035707 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:25.072165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:25.072132 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" event={"ID":"80ba56da-970e-42e4-9319-c053cd9e76b9","Type":"ContainerStarted","Data":"54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a"} Oct 29 23:32:25.072296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:25.072203 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:25.072296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:25.072228 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" event={"ID":"80ba56da-970e-42e4-9319-c053cd9e76b9","Type":"ContainerStarted","Data":"b73b4f0d5ba5e93f14e3951e9ab8a785894aad9ddc076a90b7be3465a56dc784"} Oct 29 23:32:25.111843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:25.111727 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" podStartSLOduration=1.11170776 podStartE2EDuration="1.11170776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:25.109252319 +0000 UTC m=+2942.225171925" watchObservedRunningTime="2026-10-29 23:32:25.11170776 +0000 UTC m=+2942.227627364" Oct 29 23:32:25.355994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:25.355955 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl"] Oct 29 23:32:25.623632 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:25.623518 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e2ac7/agc-cred-366eb7011dee: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:25.623632 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:25.623552 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:25.623853 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:25.623667 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials podName:bf362d65-c122-473d-9359-64d294b3a535 nodeName:}" failed. No retries permitted until 2026-10-29 23:32:27.623646153 +0000 UTC m=+2944.739565755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" (UID: "bf362d65-c122-473d-9359-64d294b3a535") : secret "agc-cred-366eb7011dee" not found Oct 29 23:32:26.081457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:26.081414 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" event={"ID":"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1","Type":"ContainerStarted","Data":"5b2e2db7688392c66614a1a4f0e4d00aa78d4e91468928f59138ee92b0303fe4"} Oct 29 23:32:26.081953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:26.081468 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" event={"ID":"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1","Type":"ContainerStarted","Data":"f2107918e8cb88793f94e2410ebae7ced920c479e0b8b5829736cc95145b24f6"} Oct 29 23:32:26.124300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:26.124251 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" podStartSLOduration=2.124236898 podStartE2EDuration="2.124236898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:26.117554299 +0000 UTC m=+2943.233473905" watchObservedRunningTime="2026-10-29 23:32:26.124236898 +0000 UTC m=+2943.240156504" Oct 29 23:32:26.824604 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:26.824554 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:32:27.091903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:27.091809 5736 generic.go:356] "Generic (PLEG): container finished" podID="95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1" containerID="5b2e2db7688392c66614a1a4f0e4d00aa78d4e91468928f59138ee92b0303fe4" exitCode=0 Oct 29 23:32:27.091903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:27.091882 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" event={"ID":"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1","Type":"ContainerDied","Data":"5b2e2db7688392c66614a1a4f0e4d00aa78d4e91468928f59138ee92b0303fe4"} Oct 29 23:32:27.654633 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:27.654608 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e2ac7/agc-cred-366eb7011dee: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:27.654755 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:27.654639 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:27.654755 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:27.654718 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials podName:bf362d65-c122-473d-9359-64d294b3a535 nodeName:}" failed. No retries permitted until 2026-10-29 23:32:31.654697583 +0000 UTC m=+2948.770617170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" (UID: "bf362d65-c122-473d-9359-64d294b3a535") : secret "agc-cred-366eb7011dee" not found Oct 29 23:32:28.371068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.371005 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:28.425362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.425320 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-xks-ca\") pod \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " Oct 29 23:32:28.425540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.425517 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-openshift-ca\") pod \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " Oct 29 23:32:28.425686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.425668 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-kube-api-access-p95rq\" (UniqueName: \"kubernetes.io/projected/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-kube-api-access-p95rq\") pod \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\" (UID: \"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1\") " Oct 29 23:32:28.426009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.425980 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-openshift-ca" pod "95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1" (UID: "95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:28.426450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.426427 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:28.428283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.428253 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-kube-api-access-p95rq" pod "95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1" (UID: "95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1"). InnerVolumeSpecName "kube-api-access-p95rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:28.435081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.435049 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-xks-ca" pod "95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1" (UID: "95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:28.528878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.528847 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:28.528878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:28.528875 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p95rq\" (UniqueName: \"kubernetes.io/projected/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1-kube-api-access-p95rq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:29.118321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.109884 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" event={"ID":"95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1","Type":"ContainerDied","Data":"f2107918e8cb88793f94e2410ebae7ced920c479e0b8b5829736cc95145b24f6"} Oct 29 23:32:29.118321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.109997 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2107918e8cb88793f94e2410ebae7ced920c479e0b8b5829736cc95145b24f6" Oct 29 23:32:29.118321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.110196 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl" Oct 29 23:32:29.130164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.130112 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" podUID="abc9d562-8ded-454e-83f8-0008f4566d15" containerName="payload-processing" containerID="cri-o://88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929" gracePeriod=25 Oct 29 23:32:29.434335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.434310 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:32:29.543347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.543318 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-provider-credentials\") pod \"abc9d562-8ded-454e-83f8-0008f4566d15\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " Oct 29 23:32:29.543508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.543427 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-plugins-config-volume\") pod \"abc9d562-8ded-454e-83f8-0008f4566d15\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " Oct 29 23:32:29.543508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.543459 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-routing-overlay-volume\") pod \"abc9d562-8ded-454e-83f8-0008f4566d15\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " Oct 29 23:32:29.543611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.543546 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-kube-api-access-98sjc\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-kube-api-access-98sjc\") pod \"abc9d562-8ded-454e-83f8-0008f4566d15\" (UID: \"abc9d562-8ded-454e-83f8-0008f4566d15\") " Oct 29 23:32:29.543918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.543889 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-routing-overlay-volume" pod "abc9d562-8ded-454e-83f8-0008f4566d15" (UID: "abc9d562-8ded-454e-83f8-0008f4566d15"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:29.544022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.543918 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-plugins-config-volume" pod "abc9d562-8ded-454e-83f8-0008f4566d15" (UID: "abc9d562-8ded-454e-83f8-0008f4566d15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:29.544190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.544173 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:29.544232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.544192 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc9d562-8ded-454e-83f8-0008f4566d15-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:29.545475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.545455 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-provider-credentials" pod "abc9d562-8ded-454e-83f8-0008f4566d15" (UID: "abc9d562-8ded-454e-83f8-0008f4566d15"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:29.545676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.545655 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-kube-api-access-98sjc" pod "abc9d562-8ded-454e-83f8-0008f4566d15" (UID: "abc9d562-8ded-454e-83f8-0008f4566d15"). InnerVolumeSpecName "kube-api-access-98sjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:29.645476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.645446 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98sjc\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-kube-api-access-98sjc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:29.645476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:29.645473 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc9d562-8ded-454e-83f8-0008f4566d15-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:30.167730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:30.167694 5736 generic.go:356] "Generic (PLEG): container finished" podID="abc9d562-8ded-454e-83f8-0008f4566d15" containerID="88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929" exitCode=0 Oct 29 23:32:30.167896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:30.167777 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" event={"ID":"abc9d562-8ded-454e-83f8-0008f4566d15","Type":"ContainerDied","Data":"88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929"} Oct 29 23:32:30.167896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:30.167816 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" Oct 29 23:32:30.167896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:30.167838 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz" event={"ID":"abc9d562-8ded-454e-83f8-0008f4566d15","Type":"ContainerDied","Data":"b0b80003c3dd8c3e14ba3bef9132966a12d6ac59ca334ccaf17726d3f69fec9c"} Oct 29 23:32:30.167896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:30.167865 5736 scope.go:122] "RemoveContainer" containerID="88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929" Oct 29 23:32:30.181668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:30.181646 5736 scope.go:122] "RemoveContainer" containerID="88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929" Oct 29 23:32:30.181953 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:30.181928 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929\": container with ID starting with 88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929 not found: ID does not exist" containerID="88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929" Oct 29 23:32:30.182036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:30.181961 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929"} err="failed to get container status \"88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929\": rpc error: code = NotFound desc = could not find container \"88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929\": container with ID starting with 88d22a6723f48ec393dab004ffec51f116f421ccf7e4afbfb68a024b314df929 not found: ID does not exist" Oct 29 23:32:30.217728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:30.217704 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz"] Oct 29 23:32:30.226977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:30.226952 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-64d9t7rzz"] Oct 29 23:32:31.005851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.005806 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc9d562-8ded-454e-83f8-0008f4566d15" path="/var/lib/kubelet/pods/abc9d562-8ded-454e-83f8-0008f4566d15/volumes" Oct 29 23:32:31.638698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.638666 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls"] Oct 29 23:32:31.639536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.639513 5736 cpu_manager.go:414] "containerMap: removing container" podUID="abc9d562-8ded-454e-83f8-0008f4566d15" containerName="payload-processing" Oct 29 23:32:31.639536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.639536 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc9d562-8ded-454e-83f8-0008f4566d15" containerName="payload-processing" Oct 29 23:32:31.639770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.639636 5736 cpu_manager.go:414] "containerMap: removing container" podUID="95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1" containerName="revoke-keys" Oct 29 23:32:31.639770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.639647 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1" containerName="revoke-keys" Oct 29 23:32:31.639881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.639854 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1" containerName="revoke-keys" Oct 29 23:32:31.639942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.639879 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc9d562-8ded-454e-83f8-0008f4566d15" containerName="payload-processing" Oct 29 23:32:31.644988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.644966 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.665616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.665560 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls"] Oct 29 23:32:31.670861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.670835 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5cd0f2-serving-cert\"" Oct 29 23:32:31.677010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.676974 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9da55-34af-4cde-8c11-5427d6e81393-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.677143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.677118 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.677259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.677239 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jk5\" (UniqueName: \"kubernetes.io/projected/5ab9da55-34af-4cde-8c11-5427d6e81393-kube-api-access-p6jk5\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.677701 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:31.677685 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e2ac7/agc-cred-366eb7011dee: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:31.677805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.677781 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-metrics-certs\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.677805 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:31.677795 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:31.678044 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:31.678030 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials podName:bf362d65-c122-473d-9359-64d294b3a535 nodeName:}" failed. No retries permitted until 2026-10-29 23:32:39.678010956 +0000 UTC m=+2956.793930547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" (UID: "bf362d65-c122-473d-9359-64d294b3a535") : secret "agc-cred-366eb7011dee" not found Oct 29 23:32:31.785691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.785657 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jk5\" (UniqueName: \"kubernetes.io/projected/5ab9da55-34af-4cde-8c11-5427d6e81393-kube-api-access-p6jk5\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.785880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.785754 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-metrics-certs\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.785958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.785895 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9da55-34af-4cde-8c11-5427d6e81393-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.786027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.785982 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.786095 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:31.786065 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-serving-cert: secret "maas-api-e2e-inf-a-5cd0f2-serving-cert" not found Oct 29 23:32:31.786156 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:31.786144 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls podName:5ab9da55-34af-4cde-8c11-5427d6e81393 nodeName:}" failed. No retries permitted until 2026-10-29 23:32:32.286124837 +0000 UTC m=+2949.402044436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls") pod "maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" (UID: "5ab9da55-34af-4cde-8c11-5427d6e81393") : secret "maas-api-e2e-inf-a-5cd0f2-serving-cert" not found Oct 29 23:32:31.786563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.786544 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9da55-34af-4cde-8c11-5427d6e81393-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.788145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.788126 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-metrics-certs\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:31.808954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:31.808929 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jk5\" (UniqueName: \"kubernetes.io/projected/5ab9da55-34af-4cde-8c11-5427d6e81393-kube-api-access-p6jk5\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:32.296381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:32.296339 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:32.298958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:32.298931 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls\") pod \"maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:32.559400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:32.559319 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:33.073201 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:32:33.073129 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab9da55_34af_4cde_8c11_5427d6e81393.slice/crio-7d4aef634436263446b4ad5feeff91b4eba143d56a18dff1555ac6d2f8894083 WatchSource:0}: Error finding container 7d4aef634436263446b4ad5feeff91b4eba143d56a18dff1555ac6d2f8894083: Status 404 returned error can't find the container with id 7d4aef634436263446b4ad5feeff91b4eba143d56a18dff1555ac6d2f8894083 Oct 29 23:32:33.086589 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:33.086546 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls"] Oct 29 23:32:33.196342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:33.196247 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" event={"ID":"5ab9da55-34af-4cde-8c11-5427d6e81393","Type":"ContainerStarted","Data":"7d4aef634436263446b4ad5feeff91b4eba143d56a18dff1555ac6d2f8894083"} Oct 29 23:32:34.031019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:34.030983 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8"] Oct 29 23:32:34.040755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:34.040727 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859b17bb-q6nf8"] Oct 29 23:32:34.206648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:34.206612 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" event={"ID":"5ab9da55-34af-4cde-8c11-5427d6e81393","Type":"ContainerStarted","Data":"798fe8d02555147af8e4e2abab4af9ea39c39679106a87b77259cf1eb44f9037"} Oct 29 23:32:34.206850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:34.206829 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:34.247378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:34.247332 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" podStartSLOduration=2.877621336 podStartE2EDuration="3.247319427s" totalImagesPullingTime="369.698091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:34.245609558 +0000 UTC m=+2951.361529161" watchObservedRunningTime="2026-10-29 23:32:34.247319427 +0000 UTC m=+2951.363239033" Oct 29 23:32:35.006378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:35.006342 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33360daf-fb50-40fa-9e9f-67f16e31240f" path="/var/lib/kubelet/pods/33360daf-fb50-40fa-9e9f-67f16e31240f/volumes" Oct 29 23:32:36.084761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:36.084724 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:36.159901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:36.159860 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l"] Oct 29 23:32:36.512057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:36.512021 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr"] Oct 29 23:32:39.733957 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:39.733927 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e2ac7/agc-cred-366eb7011dee: secret "agc-cred-366eb7011dee" not found Oct 29 23:32:39.733957 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:39.733952 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e2ac7/agc-cred-68806e843198: secret "agc-cred-68806e843198" not found Oct 29 23:32:39.733957 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:39.733962 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l: [secret "agc-cred-366eb7011dee" not found, secret "agc-cred-68806e843198" not found] Oct 29 23:32:39.734526 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:39.734018 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials podName:bf362d65-c122-473d-9359-64d294b3a535 nodeName:}" failed. No retries permitted until 2026-10-29 23:32:55.733999979 +0000 UTC m=+2972.849919565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" (UID: "bf362d65-c122-473d-9359-64d294b3a535") : [secret "agc-cred-366eb7011dee" not found, secret "agc-cred-68806e843198" not found] Oct 29 23:32:40.221321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:40.221290 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:32:41.160531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.160489 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" podUID="bf362d65-c122-473d-9359-64d294b3a535" containerName="payload-processing" containerID="cri-o://0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1" gracePeriod=25 Oct 29 23:32:41.449475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.449450 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:41.512345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.512302 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" podUID="80ba56da-970e-42e4-9319-c053cd9e76b9" containerName="payload-processing" containerID="cri-o://54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a" gracePeriod=25 Oct 29 23:32:41.572293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.572264 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-plugins-config-volume\") pod \"bf362d65-c122-473d-9359-64d294b3a535\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " Oct 29 23:32:41.572405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.572348 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-kube-api-access-74krc\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-kube-api-access-74krc\") pod \"bf362d65-c122-473d-9359-64d294b3a535\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " Oct 29 23:32:41.572470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.572457 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials\") pod \"bf362d65-c122-473d-9359-64d294b3a535\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " Oct 29 23:32:41.572538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.572515 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-routing-overlay-volume\") pod \"bf362d65-c122-473d-9359-64d294b3a535\" (UID: \"bf362d65-c122-473d-9359-64d294b3a535\") " Oct 29 23:32:41.572685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.572654 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-plugins-config-volume" pod "bf362d65-c122-473d-9359-64d294b3a535" (UID: "bf362d65-c122-473d-9359-64d294b3a535"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:41.572963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.572938 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-routing-overlay-volume" pod "bf362d65-c122-473d-9359-64d294b3a535" (UID: "bf362d65-c122-473d-9359-64d294b3a535"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:41.573453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.573431 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:41.573543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.573461 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf362d65-c122-473d-9359-64d294b3a535-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:41.574515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.574492 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials" pod "bf362d65-c122-473d-9359-64d294b3a535" (UID: "bf362d65-c122-473d-9359-64d294b3a535"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:41.574610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.574561 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-kube-api-access-74krc" pod "bf362d65-c122-473d-9359-64d294b3a535" (UID: "bf362d65-c122-473d-9359-64d294b3a535"). InnerVolumeSpecName "kube-api-access-74krc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:41.675267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.675176 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74krc\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-kube-api-access-74krc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:41.675267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.675215 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf362d65-c122-473d-9359-64d294b3a535-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:41.831489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.831462 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:41.878756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.878727 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-kube-api-access-w4bd5\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-kube-api-access-w4bd5\") pod \"80ba56da-970e-42e4-9319-c053cd9e76b9\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " Oct 29 23:32:41.878912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.878848 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-plugins-config-volume\") pod \"80ba56da-970e-42e4-9319-c053cd9e76b9\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " Oct 29 23:32:41.879021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.879007 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-routing-overlay-volume\") pod \"80ba56da-970e-42e4-9319-c053cd9e76b9\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " Oct 29 23:32:41.879107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.879091 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-provider-credentials\") pod \"80ba56da-970e-42e4-9319-c053cd9e76b9\" (UID: \"80ba56da-970e-42e4-9319-c053cd9e76b9\") " Oct 29 23:32:41.879329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.879301 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-plugins-config-volume" pod "80ba56da-970e-42e4-9319-c053cd9e76b9" (UID: "80ba56da-970e-42e4-9319-c053cd9e76b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:41.879406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.879318 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-routing-overlay-volume" pod "80ba56da-970e-42e4-9319-c053cd9e76b9" (UID: "80ba56da-970e-42e4-9319-c053cd9e76b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:41.879792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.879762 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:41.879873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.879796 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ba56da-970e-42e4-9319-c053cd9e76b9-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:41.881063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.881030 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-kube-api-access-w4bd5" pod "80ba56da-970e-42e4-9319-c053cd9e76b9" (UID: "80ba56da-970e-42e4-9319-c053cd9e76b9"). InnerVolumeSpecName "kube-api-access-w4bd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:41.881179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.881075 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-provider-credentials" pod "80ba56da-970e-42e4-9319-c053cd9e76b9" (UID: "80ba56da-970e-42e4-9319-c053cd9e76b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:41.981746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.981716 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:41.981746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:41.981745 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4bd5\" (UniqueName: \"kubernetes.io/projected/80ba56da-970e-42e4-9319-c053cd9e76b9-kube-api-access-w4bd5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:32:42.274818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.274739 5736 generic.go:356] "Generic (PLEG): container finished" podID="80ba56da-970e-42e4-9319-c053cd9e76b9" containerID="54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a" exitCode=0 Oct 29 23:32:42.274818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.274787 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" Oct 29 23:32:42.275324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.274828 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" event={"ID":"80ba56da-970e-42e4-9319-c053cd9e76b9","Type":"ContainerDied","Data":"54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a"} Oct 29 23:32:42.275324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.274885 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr" event={"ID":"80ba56da-970e-42e4-9319-c053cd9e76b9","Type":"ContainerDied","Data":"b73b4f0d5ba5e93f14e3951e9ab8a785894aad9ddc076a90b7be3465a56dc784"} Oct 29 23:32:42.275324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.274912 5736 scope.go:122] "RemoveContainer" containerID="54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a" Oct 29 23:32:42.276849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.276823 5736 generic.go:356] "Generic (PLEG): container finished" podID="bf362d65-c122-473d-9359-64d294b3a535" containerID="0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1" exitCode=0 Oct 29 23:32:42.276951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.276888 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" Oct 29 23:32:42.277000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.276889 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" event={"ID":"bf362d65-c122-473d-9359-64d294b3a535","Type":"ContainerDied","Data":"0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1"} Oct 29 23:32:42.277049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.277005 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l" event={"ID":"bf362d65-c122-473d-9359-64d294b3a535","Type":"ContainerDied","Data":"adc02560958397431bcfb94fee8762cb7c6e1d3e4db33a300269958331de6e99"} Oct 29 23:32:42.288457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.288439 5736 scope.go:122] "RemoveContainer" containerID="54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a" Oct 29 23:32:42.288737 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:42.288716 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a\": container with ID starting with 54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a not found: ID does not exist" containerID="54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a" Oct 29 23:32:42.288808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.288742 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a"} err="failed to get container status \"54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a\": rpc error: code = NotFound desc = could not find container \"54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a\": container with ID starting with 54fe02f37b8e9fdabc261cb772b2b641d13698c7887173fbf4175cd669db6f3a not found: ID does not exist" Oct 29 23:32:42.288808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.288757 5736 scope.go:122] "RemoveContainer" containerID="0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1" Oct 29 23:32:42.298721 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.298702 5736 scope.go:122] "RemoveContainer" containerID="0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1" Oct 29 23:32:42.298997 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:42.298977 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1\": container with ID starting with 0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1 not found: ID does not exist" containerID="0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1" Oct 29 23:32:42.299052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.299005 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1"} err="failed to get container status \"0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1\": rpc error: code = NotFound desc = could not find container \"0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1\": container with ID starting with 0635783278b6041ac8854270c0862a1483165fcfc1d2fd75a5ec1b9c006166d1 not found: ID does not exist" Oct 29 23:32:42.325821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.325787 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr"] Oct 29 23:32:42.330426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.330405 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7c94lf5mr"] Oct 29 23:32:42.380616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.361567 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l"] Oct 29 23:32:42.380616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:42.372241 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-7cc77cr2l"] Oct 29 23:32:43.005614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:43.005567 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ba56da-970e-42e4-9319-c053cd9e76b9" path="/var/lib/kubelet/pods/80ba56da-970e-42e4-9319-c053cd9e76b9/volumes" Oct 29 23:32:43.006052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:43.006036 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf362d65-c122-473d-9359-64d294b3a535" path="/var/lib/kubelet/pods/bf362d65-c122-473d-9359-64d294b3a535/volumes" Oct 29 23:32:46.770605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:46.766730 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:32:53.590953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.590897 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk"] Oct 29 23:32:53.592801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.592767 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bf362d65-c122-473d-9359-64d294b3a535" containerName="payload-processing" Oct 29 23:32:53.592981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.592965 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf362d65-c122-473d-9359-64d294b3a535" containerName="payload-processing" Oct 29 23:32:53.593183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.593162 5736 cpu_manager.go:414] "containerMap: removing container" podUID="80ba56da-970e-42e4-9319-c053cd9e76b9" containerName="payload-processing" Oct 29 23:32:53.593277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.593265 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ba56da-970e-42e4-9319-c053cd9e76b9" containerName="payload-processing" Oct 29 23:32:53.593645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.593625 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ba56da-970e-42e4-9319-c053cd9e76b9" containerName="payload-processing" Oct 29 23:32:53.593825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.593809 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf362d65-c122-473d-9359-64d294b3a535" containerName="payload-processing" Oct 29 23:32:53.601221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.601197 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.604635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.604608 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk"] Oct 29 23:32:53.619439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.618832 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"openshift-service-ca.crt\"" Oct 29 23:32:53.625697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.625662 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"routing-overlay\"" Oct 29 23:32:53.626084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.625767 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"agc-cred-48f0e2510771\"" Oct 29 23:32:53.626084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.625791 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"kube-root-ca.crt\"" Oct 29 23:32:53.626084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.625856 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"payload-processing-external-model-e2e--50b13d0a-dockercfg-clrtk\"" Oct 29 23:32:53.626084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.625875 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a84df8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a84df8\"" Oct 29 23:32:53.759068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.754387 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.772234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.772199 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.772567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.772546 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.772972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.772952 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48zb\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-kube-api-access-g48zb\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.880864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.876698 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.880864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.876824 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.880864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.876880 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48zb\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-kube-api-access-g48zb\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.880864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.876924 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.880864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.878120 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.882164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.881726 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.931219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.931147 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48zb\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-kube-api-access-g48zb\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:53.936485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:53.936258 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:54.225465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.225080 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:54.331842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.331808 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb"] Oct 29 23:32:54.339441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.337977 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.372710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.368742 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"kube-root-ca.crt\"" Oct 29 23:32:54.372710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.368833 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4e2ac7\"" Oct 29 23:32:54.372710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.368880 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"routing-overlay\"" Oct 29 23:32:54.372710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.368986 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"payload-processing-external-model-e2e--a44b4792-dockercfg-n954g\"" Oct 29 23:32:54.372710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.369153 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"agc-cred-5d8defbe8a5e\"" Oct 29 23:32:54.372710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.370128 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e2ac7\"/\"openshift-service-ca.crt\"" Oct 29 23:32:54.383209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.383141 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb"] Oct 29 23:32:54.633012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.632971 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.633702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.633670 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.634039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.634002 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86ns\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-kube-api-access-w86ns\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.634330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.634306 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.738225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.737284 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.738225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.737362 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.738225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.737574 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86ns\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-kube-api-access-w86ns\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.738225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.737674 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.741274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.739918 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.742433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.742375 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.743481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.743434 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.763367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.761892 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86ns\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-kube-api-access-w86ns\") pod \"payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:54.903946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.903738 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk"] Oct 29 23:32:54.907004 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:32:54.906966 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7eb2bc1_ff1e_4378_9e92_f6d120533f81.slice/crio-b036336974282e3444d49a0138077f0473de3797cd73ddf5d17212012eb21b75 WatchSource:0}: Error finding container b036336974282e3444d49a0138077f0473de3797cd73ddf5d17212012eb21b75: Status 404 returned error can't find the container with id b036336974282e3444d49a0138077f0473de3797cd73ddf5d17212012eb21b75 Oct 29 23:32:54.989199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:54.989170 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:55.303505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:55.303473 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb"] Oct 29 23:32:55.305376 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:32:55.305347 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d8f67a_d528_4007_b234_dce4411fa198.slice/crio-cec243e7a5af48ae4636905209bb126227f8bd23450d2227796db1e861a8258c WatchSource:0}: Error finding container cec243e7a5af48ae4636905209bb126227f8bd23450d2227796db1e861a8258c: Status 404 returned error can't find the container with id cec243e7a5af48ae4636905209bb126227f8bd23450d2227796db1e861a8258c Oct 29 23:32:55.659145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:55.659042 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" event={"ID":"58d8f67a-d528-4007-b234-dce4411fa198","Type":"ContainerStarted","Data":"528efff55020c67ef6369ca36490c62957beff32e058df1e4e9b1079d0c6c06e"} Oct 29 23:32:55.659145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:55.659117 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" event={"ID":"58d8f67a-d528-4007-b234-dce4411fa198","Type":"ContainerStarted","Data":"cec243e7a5af48ae4636905209bb126227f8bd23450d2227796db1e861a8258c"} Oct 29 23:32:55.659702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:55.659159 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:32:55.660994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:55.660970 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" event={"ID":"b7eb2bc1-ff1e-4378-9e92-f6d120533f81","Type":"ContainerStarted","Data":"2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e"} Oct 29 23:32:55.661135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:55.661012 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" event={"ID":"b7eb2bc1-ff1e-4378-9e92-f6d120533f81","Type":"ContainerStarted","Data":"b036336974282e3444d49a0138077f0473de3797cd73ddf5d17212012eb21b75"} Oct 29 23:32:55.661135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:55.661042 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:32:55.697404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:55.697363 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" podStartSLOduration=1.69735385 podStartE2EDuration="1.69735385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:55.696787372 +0000 UTC m=+2972.812706979" watchObservedRunningTime="2026-10-29 23:32:55.69735385 +0000 UTC m=+2972.813273455" Oct 29 23:32:55.738749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:55.738705 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" podStartSLOduration=2.7386914239999998 podStartE2EDuration="2.738691424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:55.737115481 +0000 UTC m=+2972.853035098" watchObservedRunningTime="2026-10-29 23:32:55.738691424 +0000 UTC m=+2972.854611030" Oct 29 23:32:56.825884 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:56.825844 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:32:56.989406 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:32:56.989369 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d0e8aa_75af_49c5_bc60_043e2d3a16df.slice/crio-conmon-0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:32:57.683076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:57.683044 5736 generic.go:356] "Generic (PLEG): container finished" podID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerID="0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1" exitCode=0 Oct 29 23:32:57.683292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:57.683118 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" event={"ID":"60d0e8aa-75af-49c5-bc60-043e2d3a16df","Type":"ContainerDied","Data":"0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1"} Oct 29 23:32:57.683547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:57.683530 5736 scope.go:122] "RemoveContainer" containerID="0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1" Oct 29 23:32:57.685469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:57.685436 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:32:57.685598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:57.685526 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:32:57.685676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:57.685613 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:32:58.693650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:58.693611 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" event={"ID":"60d0e8aa-75af-49c5-bc60-043e2d3a16df","Type":"ContainerStarted","Data":"181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25"} Oct 29 23:32:58.975978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:58.975935 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:32:58.977111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:58.977078 5736 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:32:58.977287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:58.977134 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Oct 29 23:32:59.223083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:59.223050 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:32:59.975966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:59.975928 5736 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:32:59.976403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:32:59.975984 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Oct 29 23:33:00.494931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.494892 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9"] Oct 29 23:33:00.501493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.501467 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.516520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.516489 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bddf\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-kube-api-access-9bddf\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.525384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.525356 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9"] Oct 29 23:33:00.525750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.525665 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-token\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.525948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.525930 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istiod-ca-cert\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.526096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.526082 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-envoy\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.526311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.526296 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-podinfo\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.526530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.526515 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-certs\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.526834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.526817 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-credential-socket\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.527038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.527024 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-socket\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.527409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.527391 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-data\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.528308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.528272 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9965bf-openshift-default-dockercfg-lnf8m\"" Oct 29 23:33:00.685604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.671942 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-token\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.685940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.685907 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istiod-ca-cert\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.686087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.686070 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-envoy\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.686295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.686278 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-podinfo\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.686543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.686454 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-certs\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.693821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.689455 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-credential-socket\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.694131 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.694107 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-socket\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.700812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.700783 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-data\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.701062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.701042 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bddf\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-kube-api-access-9bddf\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.718823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.717425 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-socket\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.718823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.717787 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-credential-socket\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.718823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.718058 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-data\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.719800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.719775 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istiod-ca-cert\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.771645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.770924 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-certs\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.798161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.796567 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-token\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.807613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.804338 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bddf\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-kube-api-access-9bddf\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.844555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.843764 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-podinfo\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.844555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.843976 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-envoy\") pod \"e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:00.984478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.983981 5736 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:33:00.984478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:00.984043 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Oct 29 23:33:01.129677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.129562 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:01.552171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.552128 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9"] Oct 29 23:33:01.564887 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:33:01.564847 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecabf71e_aeb5_4a40_a591_1dfbb4f0468e.slice/crio-320422ec1f390a1a800998ae5de7ae4419bdc495ec15bcd0a512b49edc6f1717 WatchSource:0}: Error finding container 320422ec1f390a1a800998ae5de7ae4419bdc495ec15bcd0a512b49edc6f1717: Status 404 returned error can't find the container with id 320422ec1f390a1a800998ae5de7ae4419bdc495ec15bcd0a512b49edc6f1717 Oct 29 23:33:01.569156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.568695 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:33:01.569156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.568763 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:33:01.569156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.568794 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:33:01.811086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.811005 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" event={"ID":"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e","Type":"ContainerStarted","Data":"9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd"} Oct 29 23:33:01.812507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.812476 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" event={"ID":"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e","Type":"ContainerStarted","Data":"320422ec1f390a1a800998ae5de7ae4419bdc495ec15bcd0a512b49edc6f1717"} Oct 29 23:33:01.864303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.864244 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podStartSLOduration=1.864225272 podStartE2EDuration="1.864225272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:01.86192677 +0000 UTC m=+2978.977846380" watchObservedRunningTime="2026-10-29 23:33:01.864225272 +0000 UTC m=+2978.980144880" Oct 29 23:33:01.995260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.995184 5736 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:33:01.995260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:01.995243 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Oct 29 23:33:02.130407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:02.130318 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:02.133723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:02.133689 5736 patch_prober.go:29] interesting pod/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" start-of-body= Oct 29 23:33:02.133970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:02.133943 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" Oct 29 23:33:02.976058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:02.976022 5736 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:33:02.976279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:02.976074 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Oct 29 23:33:03.156330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:03.155958 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:03.855775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:03.843638 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:03.856523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:03.856498 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:33:03.976675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:03.976604 5736 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Oct 29 23:33:03.976675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:03.976665 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Oct 29 23:33:04.981710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:04.981682 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:33:04.982805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:04.982782 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:33:06.675402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:06.675372 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:33:06.675943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:06.675447 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:33:19.642842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.642801 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh"] Oct 29 23:33:19.648861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.648840 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.673793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.673766 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-51462d-serving-cert\"" Oct 29 23:33:19.678889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.678862 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh"] Oct 29 23:33:19.754448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.754407 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.754642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.754593 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz9b5\" (UniqueName: \"kubernetes.io/projected/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-kube-api-access-qz9b5\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.754710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.754659 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-maas-api-tls\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.754710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.754693 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-metrics-certs\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.855787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.855755 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.855947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.855832 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz9b5\" (UniqueName: \"kubernetes.io/projected/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-kube-api-access-qz9b5\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.855947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.855859 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-maas-api-tls\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.855947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.855878 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-metrics-certs\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.856428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.856405 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.858406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.858383 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-maas-api-tls\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.858766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.858743 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-metrics-certs\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.877949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.877924 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz9b5\" (UniqueName: \"kubernetes.io/projected/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-kube-api-access-qz9b5\") pod \"maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:19.967384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:19.965985 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:20.295529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:20.295498 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh"] Oct 29 23:33:20.296366 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:33:20.296335 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de3e4a7_4aae_4765_b89b_8795c01ac9d2.slice/crio-5d6abbbb05b7847676c024a56c8b711465b5260e562e917707fda9577c5ea945 WatchSource:0}: Error finding container 5d6abbbb05b7847676c024a56c8b711465b5260e562e917707fda9577c5ea945: Status 404 returned error can't find the container with id 5d6abbbb05b7847676c024a56c8b711465b5260e562e917707fda9577c5ea945 Oct 29 23:33:21.035396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:21.035354 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" event={"ID":"5de3e4a7-4aae-4765-b89b-8795c01ac9d2","Type":"ContainerStarted","Data":"c6302f4ef4d26ea5e51d7beb48dfb7167acae0419c4f0edc1a04cac91676bf6c"} Oct 29 23:33:21.035958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:21.035420 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" event={"ID":"5de3e4a7-4aae-4765-b89b-8795c01ac9d2","Type":"ContainerStarted","Data":"5d6abbbb05b7847676c024a56c8b711465b5260e562e917707fda9577c5ea945"} Oct 29 23:33:21.035958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:21.035477 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:21.081075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:21.081015 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" podStartSLOduration=1.752756295 podStartE2EDuration="2.08099808s" totalImagesPullingTime="328.241785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:21.077134378 +0000 UTC m=+2998.193053984" watchObservedRunningTime="2026-10-29 23:33:21.08099808 +0000 UTC m=+2998.196917690" Oct 29 23:33:23.956370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:23.956338 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:33:23.986862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:23.986840 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:33:26.826569 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:33:26.826532 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:33:27.050036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:27.050006 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:33:29.781404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:29.781364 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj"] Oct 29 23:33:29.788035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:29.788011 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:29.807338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:29.807304 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj"] Oct 29 23:33:29.810401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:29.810378 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:33:29.810611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:29.810418 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:33:29.967347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:29.967300 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvdg8\" (UniqueName: \"kubernetes.io/projected/1d2bc491-821c-45ee-9068-87d35cd70334-kube-api-access-cvdg8\") pod \"maas-api-revoke-sub-4b88163e-dzgjj\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:29.967563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:29.967411 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b88163e-dzgjj\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:29.967687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:29.967627 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b88163e-dzgjj\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:30.073529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:30.073434 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvdg8\" (UniqueName: \"kubernetes.io/projected/1d2bc491-821c-45ee-9068-87d35cd70334-kube-api-access-cvdg8\") pod \"maas-api-revoke-sub-4b88163e-dzgjj\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:30.073851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:30.073825 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b88163e-dzgjj\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:30.075440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:30.075412 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b88163e-dzgjj\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:30.076083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:30.075200 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b88163e-dzgjj\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:30.093622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:30.088177 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b88163e-dzgjj\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:30.107673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:30.101344 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvdg8\" (UniqueName: \"kubernetes.io/projected/1d2bc491-821c-45ee-9068-87d35cd70334-kube-api-access-cvdg8\") pod \"maas-api-revoke-sub-4b88163e-dzgjj\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:30.133520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:30.110943 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:30.720961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:30.720925 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj"] Oct 29 23:33:30.722823 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:33:30.722787 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2bc491_821c_45ee_9068_87d35cd70334.slice/crio-cc0cc31c96e9c623c76beeee261a9f7f0984f1125e329dad05b20e93ff34bf1b WatchSource:0}: Error finding container cc0cc31c96e9c623c76beeee261a9f7f0984f1125e329dad05b20e93ff34bf1b: Status 404 returned error can't find the container with id cc0cc31c96e9c623c76beeee261a9f7f0984f1125e329dad05b20e93ff34bf1b Oct 29 23:33:31.438665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:31.438565 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" event={"ID":"1d2bc491-821c-45ee-9068-87d35cd70334","Type":"ContainerStarted","Data":"03ace19a7380268de020e75d130208976b74fdb9d870e36539397f527b4bb1f9"} Oct 29 23:33:31.438665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:31.438664 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" event={"ID":"1d2bc491-821c-45ee-9068-87d35cd70334","Type":"ContainerStarted","Data":"cc0cc31c96e9c623c76beeee261a9f7f0984f1125e329dad05b20e93ff34bf1b"} Oct 29 23:33:31.479481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:31.479422 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" podStartSLOduration=2.479404895 podStartE2EDuration="2.479404895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:31.478380819 +0000 UTC m=+3008.594300424" watchObservedRunningTime="2026-10-29 23:33:31.479404895 +0000 UTC m=+3008.595324504" Oct 29 23:33:32.448712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:32.448673 5736 generic.go:356] "Generic (PLEG): container finished" podID="1d2bc491-821c-45ee-9068-87d35cd70334" containerID="03ace19a7380268de020e75d130208976b74fdb9d870e36539397f527b4bb1f9" exitCode=0 Oct 29 23:33:32.449149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:32.448765 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" event={"ID":"1d2bc491-821c-45ee-9068-87d35cd70334","Type":"ContainerDied","Data":"03ace19a7380268de020e75d130208976b74fdb9d870e36539397f527b4bb1f9"} Oct 29 23:33:33.698761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.698728 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:33.757443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.757363 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2bc491-821c-45ee-9068-87d35cd70334-kube-api-access-cvdg8\" (UniqueName: \"kubernetes.io/projected/1d2bc491-821c-45ee-9068-87d35cd70334-kube-api-access-cvdg8\") pod \"1d2bc491-821c-45ee-9068-87d35cd70334\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " Oct 29 23:33:33.757443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.757427 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-xks-ca\") pod \"1d2bc491-821c-45ee-9068-87d35cd70334\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " Oct 29 23:33:33.757786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.757682 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-openshift-ca\") pod \"1d2bc491-821c-45ee-9068-87d35cd70334\" (UID: \"1d2bc491-821c-45ee-9068-87d35cd70334\") " Oct 29 23:33:33.759130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.759097 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-openshift-ca" pod "1d2bc491-821c-45ee-9068-87d35cd70334" (UID: "1d2bc491-821c-45ee-9068-87d35cd70334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:33.759834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.759807 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2bc491-821c-45ee-9068-87d35cd70334-kube-api-access-cvdg8" pod "1d2bc491-821c-45ee-9068-87d35cd70334" (UID: "1d2bc491-821c-45ee-9068-87d35cd70334"). InnerVolumeSpecName "kube-api-access-cvdg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:33.760065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.760041 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-xks-ca" pod "1d2bc491-821c-45ee-9068-87d35cd70334" (UID: "1d2bc491-821c-45ee-9068-87d35cd70334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:33.859780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.859744 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvdg8\" (UniqueName: \"kubernetes.io/projected/1d2bc491-821c-45ee-9068-87d35cd70334-kube-api-access-cvdg8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:33:33.859780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.859780 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:33:33.859955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:33.859809 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2bc491-821c-45ee-9068-87d35cd70334-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:33:34.472536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:34.472507 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" Oct 29 23:33:34.472743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:34.472538 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj" event={"ID":"1d2bc491-821c-45ee-9068-87d35cd70334","Type":"ContainerDied","Data":"cc0cc31c96e9c623c76beeee261a9f7f0984f1125e329dad05b20e93ff34bf1b"} Oct 29 23:33:34.472743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:33:34.472615 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc0cc31c96e9c623c76beeee261a9f7f0984f1125e329dad05b20e93ff34bf1b" Oct 29 23:33:56.827939 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:33:56.827900 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:34:17.036496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:17.036460 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5"] Oct 29 23:34:17.048037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:17.048008 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a693068-df6g5"] Oct 29 23:34:19.008694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:19.008652 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940a7468-b896-4faf-8f09-fcd83db78892" path="/var/lib/kubelet/pods/940a7468-b896-4faf-8f09-fcd83db78892/volumes" Oct 29 23:34:26.825948 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:34:26.825910 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:34:27.037806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:27.037771 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9"] Oct 29 23:34:27.049959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:27.049932 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880e1e95-5mlx9"] Oct 29 23:34:27.736635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:27.733923 5736 scope.go:122] "RemoveContainer" containerID="8079675c6e7f9b614ae62e52112cdd079db574a2dde0dffc79bf4d6b7ebe1dc2" Oct 29 23:34:29.008984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:29.008948 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b82e211-44f1-4599-b15e-07aa062fb35f" path="/var/lib/kubelet/pods/3b82e211-44f1-4599-b15e-07aa062fb35f/volumes" Oct 29 23:34:34.030504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:34.030423 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5"] Oct 29 23:34:34.042773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:34.042742 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbd544-bwqq5"] Oct 29 23:34:35.005781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:35.005750 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a87911d-6962-44f8-9307-5b8a70dab36c" path="/var/lib/kubelet/pods/9a87911d-6962-44f8-9307-5b8a70dab36c/volumes" Oct 29 23:34:41.037370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:41.037327 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv"] Oct 29 23:34:41.046789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:41.046758 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a30064-29lbv"] Oct 29 23:34:43.008495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:43.008458 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20283c0-98d0-4c3e-a5e1-8dcb47817b36" path="/var/lib/kubelet/pods/c20283c0-98d0-4c3e-a5e1-8dcb47817b36/volumes" Oct 29 23:34:46.408826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.406349 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw"] Oct 29 23:34:46.414529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.414499 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1d2bc491-821c-45ee-9068-87d35cd70334" containerName="revoke-keys" Oct 29 23:34:46.414826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.414807 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2bc491-821c-45ee-9068-87d35cd70334" containerName="revoke-keys" Oct 29 23:34:46.415256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.415239 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2bc491-821c-45ee-9068-87d35cd70334" containerName="revoke-keys" Oct 29 23:34:46.426700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.426676 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.451317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.451286 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b7a567-openshift-default-dockercfg-pms6g\"" Oct 29 23:34:46.459614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.457299 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw"] Oct 29 23:34:46.519615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.513926 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171cdd2b-1e57-45e7-81db-177626f01707-istiod-ca-cert\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.519615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.514029 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171cdd2b-1e57-45e7-81db-177626f01707-istio-podinfo\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.519615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.514073 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-istio-token\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.519615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.514246 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngsh8\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-kube-api-access-ngsh8\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.519615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.514344 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-data\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.519615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.514403 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-credential-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.519615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.514439 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-certs\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.519615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.514499 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.519615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.514531 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-envoy\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.647601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.647549 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-data\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.649461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.649429 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-credential-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.649784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.649760 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-certs\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.649966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.649950 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.650164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.650147 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-envoy\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.656241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.655792 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171cdd2b-1e57-45e7-81db-177626f01707-istiod-ca-cert\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.656477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.656454 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171cdd2b-1e57-45e7-81db-177626f01707-istio-podinfo\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.656633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.656616 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-istio-token\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.656948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.656927 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngsh8\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-kube-api-access-ngsh8\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.657540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.650877 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-credential-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.658469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.658447 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171cdd2b-1e57-45e7-81db-177626f01707-istiod-ca-cert\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.658621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.648228 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-data\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.663989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.651048 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-certs\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.665432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.651197 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-socket\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.668647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.668622 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171cdd2b-1e57-45e7-81db-177626f01707-istio-podinfo\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.694624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.684273 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-envoy\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.694852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.689475 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-istio-token\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.697407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.697379 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngsh8\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-kube-api-access-ngsh8\") pod \"e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:46.790586 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:46.790558 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:47.155057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:47.155026 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw"] Oct 29 23:34:47.158824 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:34:47.158792 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171cdd2b_1e57_45e7_81db_177626f01707.slice/crio-6893b88d85dca2ada1fc7b92989eac5f3a54eb355bacad9c46c4d135284bd3de WatchSource:0}: Error finding container 6893b88d85dca2ada1fc7b92989eac5f3a54eb355bacad9c46c4d135284bd3de: Status 404 returned error can't find the container with id 6893b88d85dca2ada1fc7b92989eac5f3a54eb355bacad9c46c4d135284bd3de Oct 29 23:34:47.163089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:47.163047 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:34:47.163216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:47.163137 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:34:47.163216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:47.163182 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:34:47.344303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:47.344267 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" event={"ID":"171cdd2b-1e57-45e7-81db-177626f01707","Type":"ContainerStarted","Data":"5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf"} Oct 29 23:34:47.344521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:47.344333 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" event={"ID":"171cdd2b-1e57-45e7-81db-177626f01707","Type":"ContainerStarted","Data":"6893b88d85dca2ada1fc7b92989eac5f3a54eb355bacad9c46c4d135284bd3de"} Oct 29 23:34:47.407403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:47.407286 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" podStartSLOduration=1.407267653 podStartE2EDuration="1.407267653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:47.397303804 +0000 UTC m=+3084.513223412" watchObservedRunningTime="2026-10-29 23:34:47.407267653 +0000 UTC m=+3084.523187259" Oct 29 23:34:47.796601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:47.790770 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:48.044127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:48.044089 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p"] Oct 29 23:34:48.054135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:48.054064 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ac438a-qz27p"] Oct 29 23:34:48.798180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:48.798146 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:49.006728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:49.006699 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b873c346-5539-42ad-926e-a92c37559eb9" path="/var/lib/kubelet/pods/b873c346-5539-42ad-926e-a92c37559eb9/volumes" Oct 29 23:34:49.361689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:49.361655 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:49.362421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:49.362400 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:34:55.032304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:55.032265 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847"] Oct 29 23:34:55.043783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:55.043751 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ffa982-hb847"] Oct 29 23:34:56.825573 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:34:56.825530 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:34:57.008664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:34:57.008625 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="693b6d84-ab38-4cc7-a9ec-55184c781679" path="/var/lib/kubelet/pods/693b6d84-ab38-4cc7-a9ec-55184c781679/volumes" Oct 29 23:35:27.081438 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:35:27.081390 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:35:27.848020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:27.847990 5736 scope.go:122] "RemoveContainer" containerID="7d6242c5355425ffbf6de37897fadf126039132c5a5a8862239b0d6cab402677" Oct 29 23:35:27.859741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:27.859559 5736 scope.go:122] "RemoveContainer" containerID="2d4ae5167948d07ed55e44d225111b2556772f361e246a3df0b27d4dcb7357b6" Oct 29 23:35:38.860807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:38.860713 5736 generic.go:356] "Generic (PLEG): container finished" podID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerID="9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd" exitCode=0 Oct 29 23:35:38.860807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:38.860789 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" event={"ID":"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e","Type":"ContainerDied","Data":"9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd"} Oct 29 23:35:38.861273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:38.861256 5736 scope.go:122] "RemoveContainer" containerID="9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd" Oct 29 23:35:38.863916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:38.863883 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:35:38.864044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:38.863975 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:35:38.864044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:38.864021 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:35:39.888988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:39.888943 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" event={"ID":"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e","Type":"ContainerStarted","Data":"ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b"} Oct 29 23:35:40.130301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:40.130267 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:35:40.130495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:40.130424 5736 patch_prober.go:29] interesting pod/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" start-of-body= Oct 29 23:35:40.130495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:40.130475 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" Oct 29 23:35:41.130383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:41.130351 5736 patch_prober.go:29] interesting pod/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" start-of-body= Oct 29 23:35:41.130869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:41.130414 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" Oct 29 23:35:42.130126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:42.130080 5736 patch_prober.go:29] interesting pod/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" start-of-body= Oct 29 23:35:42.130285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:42.130146 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" Oct 29 23:35:43.130048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:43.130010 5736 patch_prober.go:29] interesting pod/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" start-of-body= Oct 29 23:35:43.130465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:43.130088 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" Oct 29 23:35:44.134735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:44.134696 5736 patch_prober.go:29] interesting pod/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" start-of-body= Oct 29 23:35:44.135348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:44.135310 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" Oct 29 23:35:45.129990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:45.129954 5736 patch_prober.go:29] interesting pod/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" start-of-body= Oct 29 23:35:45.130174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:45.130043 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.91:15021/healthz/ready\": dial tcp 10.134.1.91:15021: connect: connection refused" Oct 29 23:35:46.135786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.135755 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:35:46.136254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.136230 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:35:46.137025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.137001 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:35:46.200026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.199992 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg"] Oct 29 23:35:46.203957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.203929 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.225536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.218686 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-90f683\"/\"kube-root-ca.crt\"" Oct 29 23:35:46.225536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.218768 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-90f683\"/\"test-model-90f683-kserve-self-signed-certs\"" Oct 29 23:35:46.225536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.218790 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-90f683\"/\"default-dockercfg-l7mf5\"" Oct 29 23:35:46.225536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.218806 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-90f683\"/\"openshift-service-ca.crt\"" Oct 29 23:35:46.231389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.231358 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg"] Oct 29 23:35:46.310686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.310646 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-model-cache\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.311059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.311041 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-dshm\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.311262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.311248 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4171d287-41c7-4c40-871b-2fd7a55d7362-tls-certs\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.311464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.311449 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4xzm\" (UniqueName: \"kubernetes.io/projected/4171d287-41c7-4c40-871b-2fd7a55d7362-kube-api-access-k4xzm\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.311680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.311662 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-home\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.311868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.311852 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-tmp-dir\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.413266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.413176 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-model-cache\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.413621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.413575 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-dshm\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.413844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.413723 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-model-cache\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.413981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.413836 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4171d287-41c7-4c40-871b-2fd7a55d7362-tls-certs\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.414199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.414175 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4xzm\" (UniqueName: \"kubernetes.io/projected/4171d287-41c7-4c40-871b-2fd7a55d7362-kube-api-access-k4xzm\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.414421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.414401 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-home\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.414617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.414599 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-tmp-dir\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.415237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.415214 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-tmp-dir\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.415949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.415924 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-home\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.416690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.416572 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-dshm\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.416796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.416773 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4171d287-41c7-4c40-871b-2fd7a55d7362-tls-certs\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.438042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.438018 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4xzm\" (UniqueName: \"kubernetes.io/projected/4171d287-41c7-4c40-871b-2fd7a55d7362-kube-api-access-k4xzm\") pod \"test-model-90f683-kserve-f485d9ff-ztspg\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:46.520602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:46.520508 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:47.057230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:47.057201 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg"] Oct 29 23:35:47.058078 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:35:47.058053 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4171d287_41c7_4c40_871b_2fd7a55d7362.slice/crio-881cf5bf179bbc299b206da6fd6888a7f40f2bff597354687fe876ab929fe4ad WatchSource:0}: Error finding container 881cf5bf179bbc299b206da6fd6888a7f40f2bff597354687fe876ab929fe4ad: Status 404 returned error can't find the container with id 881cf5bf179bbc299b206da6fd6888a7f40f2bff597354687fe876ab929fe4ad Oct 29 23:35:47.966876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:47.966830 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" event={"ID":"4171d287-41c7-4c40-871b-2fd7a55d7362","Type":"ContainerStarted","Data":"08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1"} Oct 29 23:35:47.967373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:47.966893 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" event={"ID":"4171d287-41c7-4c40-871b-2fd7a55d7362","Type":"ContainerStarted","Data":"881cf5bf179bbc299b206da6fd6888a7f40f2bff597354687fe876ab929fe4ad"} Oct 29 23:35:48.013078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:48.013029 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" podStartSLOduration=2.013015532 podStartE2EDuration="2.013015532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:48.008407319 +0000 UTC m=+3145.124326924" watchObservedRunningTime="2026-10-29 23:35:48.013015532 +0000 UTC m=+3145.128935138" Oct 29 23:35:48.521123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:48.521092 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:56.521379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:56.521341 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:56.537493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:56.537454 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:56.827073 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:35:56.826971 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:35:57.078049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:57.077966 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:35:58.031211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:58.031173 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd"] Oct 29 23:35:58.043260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:58.043227 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37617a2f-zgrcd"] Oct 29 23:35:59.006635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:35:59.006604 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2680b5-bb27-485d-81b3-74583f9e310a" path="/var/lib/kubelet/pods/5c2680b5-bb27-485d-81b3-74583f9e310a/volumes" Oct 29 23:36:13.419499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.419307 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t"] Oct 29 23:36:13.425839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.425810 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.446516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.446487 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t"] Oct 29 23:36:13.450719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.450679 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9965bf\"/\"kube-root-ca.crt\"" Oct 29 23:36:13.450873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.450787 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9965bf\"/\"openshift-service-ca.crt\"" Oct 29 23:36:13.453728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.453708 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9965bf\"/\"default-dockercfg-fbq58\"" Oct 29 23:36:13.453877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.453858 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9965bf\"/\"test-model-9965bf-kserve-self-signed-certs\"" Oct 29 23:36:13.530506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.530467 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-dshm\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.530718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.530600 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-home\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.530877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.530859 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tls-certs\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.531020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.530989 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-model-cache\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.531135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.531106 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4llx\" (UniqueName: \"kubernetes.io/projected/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-kube-api-access-k4llx\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.531188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.531179 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tmp-dir\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.633160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.633105 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-model-cache\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.633371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.633219 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4llx\" (UniqueName: \"kubernetes.io/projected/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-kube-api-access-k4llx\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.633371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.633301 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tmp-dir\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.633371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.633362 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-dshm\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.633637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.633414 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-home\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.633732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.633636 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-model-cache\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.633800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.633728 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tls-certs\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.633800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.633763 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tmp-dir\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.633912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.633833 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-home\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.635876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.635851 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-dshm\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.636347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.636328 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tls-certs\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.680712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.671376 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4llx\" (UniqueName: \"kubernetes.io/projected/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-kube-api-access-k4llx\") pod \"test-model-9965bf-kserve-68b58fdc65-8dh4t\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:13.764044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:13.763118 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:14.209374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:14.209232 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t"] Oct 29 23:36:14.214371 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:36:14.214344 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a5cbd9f_6e91_4839_abc9_cd84c9b533f5.slice/crio-972bd55fb55a8d93d96349058c9f69c2f2a0cbce0968caa2b6c5cb27c69a3de7 WatchSource:0}: Error finding container 972bd55fb55a8d93d96349058c9f69c2f2a0cbce0968caa2b6c5cb27c69a3de7: Status 404 returned error can't find the container with id 972bd55fb55a8d93d96349058c9f69c2f2a0cbce0968caa2b6c5cb27c69a3de7 Oct 29 23:36:14.235647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:14.235614 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" event={"ID":"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5","Type":"ContainerStarted","Data":"972bd55fb55a8d93d96349058c9f69c2f2a0cbce0968caa2b6c5cb27c69a3de7"} Oct 29 23:36:15.246972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:15.246931 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" event={"ID":"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5","Type":"ContainerStarted","Data":"d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5"} Oct 29 23:36:15.291952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:15.291896 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" podStartSLOduration=2.29188077 podStartE2EDuration="2.29188077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:36:15.287465046 +0000 UTC m=+3172.403384651" watchObservedRunningTime="2026-10-29 23:36:15.29188077 +0000 UTC m=+3172.407800375" Oct 29 23:36:15.763942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:15.763907 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:23.763607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:23.763561 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:23.777182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:23.777155 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:24.333721 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:24.333692 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:36:26.825051 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:36:26.825013 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:36:27.904486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:27.904453 5736 scope.go:122] "RemoveContainer" containerID="ea83a8df00534b9f56bef7cbc5211c6bd135c685d46f20e583fc8d05d266aa70" Oct 29 23:36:27.916029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:27.916008 5736 scope.go:122] "RemoveContainer" containerID="54a1ab8c688a983b2839fc51205e40b757a305fde139a55bc867de27738c9ddc" Oct 29 23:36:27.925838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:27.925816 5736 scope.go:122] "RemoveContainer" containerID="5ea263bac44a96d4b0dc93d25d1541330504fb578a4aa7955743bc9fc8e7ca31" Oct 29 23:36:27.935069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:27.935040 5736 scope.go:122] "RemoveContainer" containerID="a95ffafb294a9f01468bd91106314f78432912ca1e6e2550121061738c515a6e" Oct 29 23:36:27.944304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:27.944271 5736 scope.go:122] "RemoveContainer" containerID="16719025f5649296312b49315ef438bf588f46d81fe34061ae116f7e16f2b0ee" Oct 29 23:36:45.664605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:45.663658 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:36:45.691601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:45.676652 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:36:46.964734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:46.964673 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd"] Oct 29 23:36:46.972759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:46.972733 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:46.981555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:46.981519 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd"] Oct 29 23:36:46.989518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:46.989487 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:36:47.001472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.001448 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:36:47.058604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.041899 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m5z2\" (UniqueName: \"kubernetes.io/projected/db92548e-5f08-4922-b4a8-88282d3557ed-kube-api-access-4m5z2\") pod \"maas-api-revoke-sub-cecc986f-7z4bd\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.059137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.059105 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cecc986f-7z4bd\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.060205 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.060174 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cecc986f-7z4bd\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.161968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.161921 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cecc986f-7z4bd\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.162434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.162400 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cecc986f-7z4bd\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.162650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.162628 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5z2\" (UniqueName: \"kubernetes.io/projected/db92548e-5f08-4922-b4a8-88282d3557ed-kube-api-access-4m5z2\") pod \"maas-api-revoke-sub-cecc986f-7z4bd\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.163190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.163162 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cecc986f-7z4bd\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.164726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.164701 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cecc986f-7z4bd\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.186311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.186286 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5z2\" (UniqueName: \"kubernetes.io/projected/db92548e-5f08-4922-b4a8-88282d3557ed-kube-api-access-4m5z2\") pod \"maas-api-revoke-sub-cecc986f-7z4bd\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.291050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.290986 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:47.668919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:47.668862 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd"] Oct 29 23:36:47.683219 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:36:47.683170 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb92548e_5f08_4922_b4a8_88282d3557ed.slice/crio-28f0077c7909341e2b7004570c6a75c5889c75b19fe6501d2566139505d41678 WatchSource:0}: Error finding container 28f0077c7909341e2b7004570c6a75c5889c75b19fe6501d2566139505d41678: Status 404 returned error can't find the container with id 28f0077c7909341e2b7004570c6a75c5889c75b19fe6501d2566139505d41678 Oct 29 23:36:48.187502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.187431 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g"] Oct 29 23:36:48.192529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.192507 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.215148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.215121 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g"] Oct 29 23:36:48.295457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.295369 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15e98629-5kq2g\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.296465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.295755 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15e98629-5kq2g\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.296465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.295843 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwvqw\" (UniqueName: \"kubernetes.io/projected/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-kube-api-access-xwvqw\") pod \"maas-api-revoke-sub-15e98629-5kq2g\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.408647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.405885 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15e98629-5kq2g\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.408647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.405963 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwvqw\" (UniqueName: \"kubernetes.io/projected/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-kube-api-access-xwvqw\") pod \"maas-api-revoke-sub-15e98629-5kq2g\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.408647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.406154 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15e98629-5kq2g\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.408647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.407033 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15e98629-5kq2g\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.415605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.415537 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15e98629-5kq2g\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.447617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.443954 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwvqw\" (UniqueName: \"kubernetes.io/projected/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-kube-api-access-xwvqw\") pod \"maas-api-revoke-sub-15e98629-5kq2g\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.508139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.508103 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:48.776050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.774672 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" event={"ID":"db92548e-5f08-4922-b4a8-88282d3557ed","Type":"ContainerStarted","Data":"f8cb3749c0f3ad829dffa9348470a479a96777e4667c2738c4ba065a460bcc5c"} Oct 29 23:36:48.776050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.774745 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" event={"ID":"db92548e-5f08-4922-b4a8-88282d3557ed","Type":"ContainerStarted","Data":"28f0077c7909341e2b7004570c6a75c5889c75b19fe6501d2566139505d41678"} Oct 29 23:36:48.860602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:48.853760 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" podStartSLOduration=2.853736124 podStartE2EDuration="2.853736124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:36:48.82600476 +0000 UTC m=+3205.941924367" watchObservedRunningTime="2026-10-29 23:36:48.853736124 +0000 UTC m=+3205.969655733" Oct 29 23:36:49.167984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:49.167949 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g"] Oct 29 23:36:49.168232 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:36:49.168203 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f306b1_59ee_4f6b_951f_9f5b85d54ccc.slice/crio-f9378b505177d76a4e7074a1574450e5c45c4afb9d97700f1e465e92f9582b61 WatchSource:0}: Error finding container f9378b505177d76a4e7074a1574450e5c45c4afb9d97700f1e465e92f9582b61: Status 404 returned error can't find the container with id f9378b505177d76a4e7074a1574450e5c45c4afb9d97700f1e465e92f9582b61 Oct 29 23:36:49.787203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:49.787168 5736 generic.go:356] "Generic (PLEG): container finished" podID="db92548e-5f08-4922-b4a8-88282d3557ed" containerID="f8cb3749c0f3ad829dffa9348470a479a96777e4667c2738c4ba065a460bcc5c" exitCode=0 Oct 29 23:36:49.787694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:49.787262 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" event={"ID":"db92548e-5f08-4922-b4a8-88282d3557ed","Type":"ContainerDied","Data":"f8cb3749c0f3ad829dffa9348470a479a96777e4667c2738c4ba065a460bcc5c"} Oct 29 23:36:49.789296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:49.789261 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" event={"ID":"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc","Type":"ContainerStarted","Data":"31b9e3021ca113109584a274ee718e97850439dc7bbf04d645bce7d515a4fbe4"} Oct 29 23:36:49.789413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:49.789319 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" event={"ID":"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc","Type":"ContainerStarted","Data":"f9378b505177d76a4e7074a1574450e5c45c4afb9d97700f1e465e92f9582b61"} Oct 29 23:36:49.870189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:49.870118 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" podStartSLOduration=1.870099926 podStartE2EDuration="1.870099926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:36:49.869405559 +0000 UTC m=+3206.985325166" watchObservedRunningTime="2026-10-29 23:36:49.870099926 +0000 UTC m=+3206.986019533" Oct 29 23:36:50.801867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:50.801834 5736 generic.go:356] "Generic (PLEG): container finished" podID="c4f306b1-59ee-4f6b-951f-9f5b85d54ccc" containerID="31b9e3021ca113109584a274ee718e97850439dc7bbf04d645bce7d515a4fbe4" exitCode=0 Oct 29 23:36:50.802289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:50.801918 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" event={"ID":"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc","Type":"ContainerDied","Data":"31b9e3021ca113109584a274ee718e97850439dc7bbf04d645bce7d515a4fbe4"} Oct 29 23:36:51.028032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.028005 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:51.178894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.178817 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db92548e-5f08-4922-b4a8-88282d3557ed-kube-api-access-4m5z2\" (UniqueName: \"kubernetes.io/projected/db92548e-5f08-4922-b4a8-88282d3557ed-kube-api-access-4m5z2\") pod \"db92548e-5f08-4922-b4a8-88282d3557ed\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " Oct 29 23:36:51.179040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.178971 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-xks-ca\") pod \"db92548e-5f08-4922-b4a8-88282d3557ed\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " Oct 29 23:36:51.179155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.179138 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-openshift-ca\") pod \"db92548e-5f08-4922-b4a8-88282d3557ed\" (UID: \"db92548e-5f08-4922-b4a8-88282d3557ed\") " Oct 29 23:36:51.179562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.179529 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-openshift-ca" pod "db92548e-5f08-4922-b4a8-88282d3557ed" (UID: "db92548e-5f08-4922-b4a8-88282d3557ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:36:51.179988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.179961 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:36:51.181226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.181199 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db92548e-5f08-4922-b4a8-88282d3557ed-kube-api-access-4m5z2" pod "db92548e-5f08-4922-b4a8-88282d3557ed" (UID: "db92548e-5f08-4922-b4a8-88282d3557ed"). InnerVolumeSpecName "kube-api-access-4m5z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:36:51.181336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.181271 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-xks-ca" pod "db92548e-5f08-4922-b4a8-88282d3557ed" (UID: "db92548e-5f08-4922-b4a8-88282d3557ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:36:51.281484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.281455 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db92548e-5f08-4922-b4a8-88282d3557ed-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:36:51.281484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.281483 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m5z2\" (UniqueName: \"kubernetes.io/projected/db92548e-5f08-4922-b4a8-88282d3557ed-kube-api-access-4m5z2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:36:51.815107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.815072 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" Oct 29 23:36:51.815610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.815096 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd" event={"ID":"db92548e-5f08-4922-b4a8-88282d3557ed","Type":"ContainerDied","Data":"28f0077c7909341e2b7004570c6a75c5889c75b19fe6501d2566139505d41678"} Oct 29 23:36:51.815610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:51.815156 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28f0077c7909341e2b7004570c6a75c5889c75b19fe6501d2566139505d41678" Oct 29 23:36:52.023388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.023354 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:52.226627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.197781 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-kube-api-access-xwvqw\" (UniqueName: \"kubernetes.io/projected/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-kube-api-access-xwvqw\") pod \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " Oct 29 23:36:52.226627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.197842 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-xks-ca\") pod \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " Oct 29 23:36:52.226627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.198071 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-openshift-ca\") pod \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\" (UID: \"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc\") " Oct 29 23:36:52.251637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.248798 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-kube-api-access-xwvqw" pod "c4f306b1-59ee-4f6b-951f-9f5b85d54ccc" (UID: "c4f306b1-59ee-4f6b-951f-9f5b85d54ccc"). InnerVolumeSpecName "kube-api-access-xwvqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:36:52.251637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.249104 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-openshift-ca" pod "c4f306b1-59ee-4f6b-951f-9f5b85d54ccc" (UID: "c4f306b1-59ee-4f6b-951f-9f5b85d54ccc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:36:52.257762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.257711 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-xks-ca" pod "c4f306b1-59ee-4f6b-951f-9f5b85d54ccc" (UID: "c4f306b1-59ee-4f6b-951f-9f5b85d54ccc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:36:52.301141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.301104 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:36:52.301141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.301140 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwvqw\" (UniqueName: \"kubernetes.io/projected/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-kube-api-access-xwvqw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:36:52.301377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.301163 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:36:52.826019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.825986 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" Oct 29 23:36:52.826402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.826030 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g" event={"ID":"c4f306b1-59ee-4f6b-951f-9f5b85d54ccc","Type":"ContainerDied","Data":"f9378b505177d76a4e7074a1574450e5c45c4afb9d97700f1e465e92f9582b61"} Oct 29 23:36:52.826402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:52.826078 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9378b505177d76a4e7074a1574450e5c45c4afb9d97700f1e465e92f9582b61" Oct 29 23:36:53.035849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:53.035814 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z"] Oct 29 23:36:53.042786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:53.042753 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076055e9-sgq2z"] Oct 29 23:36:54.032132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:54.032066 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l"] Oct 29 23:36:54.042740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:54.042711 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a70739b7-2689l"] Oct 29 23:36:54.845671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:54.845638 5736 generic.go:356] "Generic (PLEG): container finished" podID="171cdd2b-1e57-45e7-81db-177626f01707" containerID="5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf" exitCode=0 Oct 29 23:36:54.845965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:54.845713 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" event={"ID":"171cdd2b-1e57-45e7-81db-177626f01707","Type":"ContainerDied","Data":"5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf"} Oct 29 23:36:54.846185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:54.846166 5736 scope.go:122] "RemoveContainer" containerID="5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf" Oct 29 23:36:54.848453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:54.848419 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:36:54.848539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:54.848516 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:36:54.848617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:54.848568 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:36:55.027724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:55.027677 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a621e5-1928-4c40-bd6c-b02ee2f82ba2" path="/var/lib/kubelet/pods/77a621e5-1928-4c40-bd6c-b02ee2f82ba2/volumes" Oct 29 23:36:55.033055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:55.032838 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc793537-95c8-4dbb-9d45-1f292f51b97a" path="/var/lib/kubelet/pods/cc793537-95c8-4dbb-9d45-1f292f51b97a/volumes" Oct 29 23:36:55.856003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:55.855962 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" event={"ID":"171cdd2b-1e57-45e7-81db-177626f01707","Type":"ContainerStarted","Data":"eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e"} Oct 29 23:36:56.790754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:56.790719 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:36:56.791197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:56.790905 5736 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Oct 29 23:36:56.791197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:56.790960 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Oct 29 23:36:56.825469 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:36:56.825431 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:36:57.791048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.791013 5736 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Oct 29 23:36:57.791428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.791067 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Oct 29 23:36:57.817525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.817490 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc"] Oct 29 23:36:57.818330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.818313 5736 cpu_manager.go:414] "containerMap: removing container" podUID="db92548e-5f08-4922-b4a8-88282d3557ed" containerName="revoke-keys" Oct 29 23:36:57.818388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.818332 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db92548e-5f08-4922-b4a8-88282d3557ed" containerName="revoke-keys" Oct 29 23:36:57.818388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.818353 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c4f306b1-59ee-4f6b-951f-9f5b85d54ccc" containerName="revoke-keys" Oct 29 23:36:57.818388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.818358 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f306b1-59ee-4f6b-951f-9f5b85d54ccc" containerName="revoke-keys" Oct 29 23:36:57.818508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.818498 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f306b1-59ee-4f6b-951f-9f5b85d54ccc" containerName="revoke-keys" Oct 29 23:36:57.818546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.818523 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="db92548e-5f08-4922-b4a8-88282d3557ed" containerName="revoke-keys" Oct 29 23:36:57.823112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.823093 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:57.861603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.859231 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:36:57.864374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.864341 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc"] Oct 29 23:36:57.868620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.868596 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:36:57.910886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.910851 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc58fbf-v2xfc\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:57.911069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.910995 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsczm\" (UniqueName: \"kubernetes.io/projected/107c1179-356a-4afb-9fe5-6b0dad4a2868-kube-api-access-gsczm\") pod \"maas-api-revoke-sub-cdc58fbf-v2xfc\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:57.911153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:57.911112 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc58fbf-v2xfc\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:58.012370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.012326 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc58fbf-v2xfc\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:58.012637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.012614 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc58fbf-v2xfc\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:58.012810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.012785 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsczm\" (UniqueName: \"kubernetes.io/projected/107c1179-356a-4afb-9fe5-6b0dad4a2868-kube-api-access-gsczm\") pod \"maas-api-revoke-sub-cdc58fbf-v2xfc\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:58.013285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.013263 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc58fbf-v2xfc\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:58.014834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.014810 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc58fbf-v2xfc\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:58.057260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.057175 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsczm\" (UniqueName: \"kubernetes.io/projected/107c1179-356a-4afb-9fe5-6b0dad4a2868-kube-api-access-gsczm\") pod \"maas-api-revoke-sub-cdc58fbf-v2xfc\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:58.169789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.169752 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:36:58.505410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.505374 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc"] Oct 29 23:36:58.509964 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:36:58.509928 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107c1179_356a_4afb_9fe5_6b0dad4a2868.slice/crio-a00e9aefac2f1cb216b35fc248a08b7ebbd844be5a1a10b6f5bd4a8797bdc8de WatchSource:0}: Error finding container a00e9aefac2f1cb216b35fc248a08b7ebbd844be5a1a10b6f5bd4a8797bdc8de: Status 404 returned error can't find the container with id a00e9aefac2f1cb216b35fc248a08b7ebbd844be5a1a10b6f5bd4a8797bdc8de Oct 29 23:36:58.562140 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.552885 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt"] Oct 29 23:36:58.571315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.569724 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.575624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.575159 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt"] Oct 29 23:36:58.623516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.623481 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6pkf\" (UniqueName: \"kubernetes.io/projected/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-kube-api-access-g6pkf\") pod \"maas-api-revoke-sub-2e5babbf-ns7dt\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.623989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.623967 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e5babbf-ns7dt\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.624572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.624160 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e5babbf-ns7dt\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.726777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.726740 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6pkf\" (UniqueName: \"kubernetes.io/projected/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-kube-api-access-g6pkf\") pod \"maas-api-revoke-sub-2e5babbf-ns7dt\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.726957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.726825 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e5babbf-ns7dt\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.726957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.726865 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e5babbf-ns7dt\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.727615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.727574 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e5babbf-ns7dt\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.729244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.729219 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e5babbf-ns7dt\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.756425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.756341 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6pkf\" (UniqueName: \"kubernetes.io/projected/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-kube-api-access-g6pkf\") pod \"maas-api-revoke-sub-2e5babbf-ns7dt\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:58.791616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.791561 5736 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Oct 29 23:36:58.791932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.791641 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Oct 29 23:36:58.888618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.888550 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" event={"ID":"107c1179-356a-4afb-9fe5-6b0dad4a2868","Type":"ContainerStarted","Data":"a00e9aefac2f1cb216b35fc248a08b7ebbd844be5a1a10b6f5bd4a8797bdc8de"} Oct 29 23:36:58.919117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:58.919088 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:36:59.306766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.306736 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt"] Oct 29 23:36:59.309598 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:36:59.309540 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ee1d8d_96b9_4fe2_9371_dac4613d4f64.slice/crio-41bae9dce33c0f5c6ae7d24cb14588d3f6b7a8e4b1c663e97a064753a0fff018 WatchSource:0}: Error finding container 41bae9dce33c0f5c6ae7d24cb14588d3f6b7a8e4b1c663e97a064753a0fff018: Status 404 returned error can't find the container with id 41bae9dce33c0f5c6ae7d24cb14588d3f6b7a8e4b1c663e97a064753a0fff018 Oct 29 23:36:59.327439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.327412 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6"] Oct 29 23:36:59.365824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.365789 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6"] Oct 29 23:36:59.365965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.365917 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.411468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.411434 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmj2f\" (UniqueName: \"kubernetes.io/projected/97427e0e-39e7-4ff2-926e-2357a1ac7b05-kube-api-access-xmj2f\") pod \"maas-api-revoke-sub-94559fe6-r27h6\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.411979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.411955 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94559fe6-r27h6\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.412135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.412120 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94559fe6-r27h6\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.514113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.514079 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94559fe6-r27h6\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.514327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.514142 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94559fe6-r27h6\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.514327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.514246 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmj2f\" (UniqueName: \"kubernetes.io/projected/97427e0e-39e7-4ff2-926e-2357a1ac7b05-kube-api-access-xmj2f\") pod \"maas-api-revoke-sub-94559fe6-r27h6\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.515109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.514897 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94559fe6-r27h6\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.516651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.516624 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94559fe6-r27h6\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.558473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.558397 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmj2f\" (UniqueName: \"kubernetes.io/projected/97427e0e-39e7-4ff2-926e-2357a1ac7b05-kube-api-access-xmj2f\") pod \"maas-api-revoke-sub-94559fe6-r27h6\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.754619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.752848 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:36:59.791170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.791132 5736 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Oct 29 23:36:59.791347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.791202 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Oct 29 23:36:59.900352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.900312 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" event={"ID":"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64","Type":"ContainerStarted","Data":"6b74cae0e513ec2b3b60cfe091f6cc44d52d694b95bbdbcf976c90d9994b6ace"} Oct 29 23:36:59.900764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.900380 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" event={"ID":"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64","Type":"ContainerStarted","Data":"41bae9dce33c0f5c6ae7d24cb14588d3f6b7a8e4b1c663e97a064753a0fff018"} Oct 29 23:36:59.902290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.902268 5736 generic.go:356] "Generic (PLEG): container finished" podID="107c1179-356a-4afb-9fe5-6b0dad4a2868" containerID="1eb65f822692f092b137800f04aa10e669327cac99cba7f9ed28014fcf121550" exitCode=0 Oct 29 23:36:59.902351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.902306 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" event={"ID":"107c1179-356a-4afb-9fe5-6b0dad4a2868","Type":"ContainerDied","Data":"1eb65f822692f092b137800f04aa10e669327cac99cba7f9ed28014fcf121550"} Oct 29 23:36:59.969782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:36:59.969717 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" podStartSLOduration=1.969698913 podStartE2EDuration="1.969698913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:36:59.964272675 +0000 UTC m=+3217.080192283" watchObservedRunningTime="2026-10-29 23:36:59.969698913 +0000 UTC m=+3217.085618518" Oct 29 23:37:00.170565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:00.170483 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6"] Oct 29 23:37:00.176702 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:37:00.176674 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97427e0e_39e7_4ff2_926e_2357a1ac7b05.slice/crio-ccfe9c4c969d34973027d126da18edffb5f1c2c3ccd2fbb0a5b1e9a3f4667e2d WatchSource:0}: Error finding container ccfe9c4c969d34973027d126da18edffb5f1c2c3ccd2fbb0a5b1e9a3f4667e2d: Status 404 returned error can't find the container with id ccfe9c4c969d34973027d126da18edffb5f1c2c3ccd2fbb0a5b1e9a3f4667e2d Oct 29 23:37:00.803602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:00.799162 5736 patch_prober.go:29] interesting pod/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Oct 29 23:37:00.803602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:00.799241 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Oct 29 23:37:00.912919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:00.912833 5736 generic.go:356] "Generic (PLEG): container finished" podID="f1ee1d8d-96b9-4fe2-9371-dac4613d4f64" containerID="6b74cae0e513ec2b3b60cfe091f6cc44d52d694b95bbdbcf976c90d9994b6ace" exitCode=0 Oct 29 23:37:00.913336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:00.912913 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" event={"ID":"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64","Type":"ContainerDied","Data":"6b74cae0e513ec2b3b60cfe091f6cc44d52d694b95bbdbcf976c90d9994b6ace"} Oct 29 23:37:00.914900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:00.914867 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" event={"ID":"97427e0e-39e7-4ff2-926e-2357a1ac7b05","Type":"ContainerStarted","Data":"12d3dc22b53907cf8e7e89978243599445203854fac3a0e620ffaaaca4908b49"} Oct 29 23:37:00.915045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:00.914936 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" event={"ID":"97427e0e-39e7-4ff2-926e-2357a1ac7b05","Type":"ContainerStarted","Data":"ccfe9c4c969d34973027d126da18edffb5f1c2c3ccd2fbb0a5b1e9a3f4667e2d"} Oct 29 23:37:01.000565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.000360 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" podStartSLOduration=2.00034622 podStartE2EDuration="2.00034622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:00.999101421 +0000 UTC m=+3218.115021028" watchObservedRunningTime="2026-10-29 23:37:01.00034622 +0000 UTC m=+3218.116265827" Oct 29 23:37:01.166072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.165995 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:37:01.272291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.272248 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-openshift-ca\") pod \"107c1179-356a-4afb-9fe5-6b0dad4a2868\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " Oct 29 23:37:01.272708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.272684 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-xks-ca\") pod \"107c1179-356a-4afb-9fe5-6b0dad4a2868\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " Oct 29 23:37:01.272788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.272759 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-openshift-ca" pod "107c1179-356a-4afb-9fe5-6b0dad4a2868" (UID: "107c1179-356a-4afb-9fe5-6b0dad4a2868"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:01.272834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.272788 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107c1179-356a-4afb-9fe5-6b0dad4a2868-kube-api-access-gsczm\" (UniqueName: \"kubernetes.io/projected/107c1179-356a-4afb-9fe5-6b0dad4a2868-kube-api-access-gsczm\") pod \"107c1179-356a-4afb-9fe5-6b0dad4a2868\" (UID: \"107c1179-356a-4afb-9fe5-6b0dad4a2868\") " Oct 29 23:37:01.273607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.273565 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:01.274998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.274971 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-xks-ca" pod "107c1179-356a-4afb-9fe5-6b0dad4a2868" (UID: "107c1179-356a-4afb-9fe5-6b0dad4a2868"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:37:01.275937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.275913 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107c1179-356a-4afb-9fe5-6b0dad4a2868-kube-api-access-gsczm" pod "107c1179-356a-4afb-9fe5-6b0dad4a2868" (UID: "107c1179-356a-4afb-9fe5-6b0dad4a2868"). InnerVolumeSpecName "kube-api-access-gsczm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:01.375620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.375572 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c1179-356a-4afb-9fe5-6b0dad4a2868-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:01.375620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.375622 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsczm\" (UniqueName: \"kubernetes.io/projected/107c1179-356a-4afb-9fe5-6b0dad4a2868-kube-api-access-gsczm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:01.796629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.796594 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:37:01.797024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.796996 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:37:01.797960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.797926 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:37:01.925559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.925527 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" Oct 29 23:37:01.925559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.925547 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc" event={"ID":"107c1179-356a-4afb-9fe5-6b0dad4a2868","Type":"ContainerDied","Data":"a00e9aefac2f1cb216b35fc248a08b7ebbd844be5a1a10b6f5bd4a8797bdc8de"} Oct 29 23:37:01.926135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.925618 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a00e9aefac2f1cb216b35fc248a08b7ebbd844be5a1a10b6f5bd4a8797bdc8de" Oct 29 23:37:01.927942 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.927908 5736 generic.go:356] "Generic (PLEG): container finished" podID="97427e0e-39e7-4ff2-926e-2357a1ac7b05" containerID="12d3dc22b53907cf8e7e89978243599445203854fac3a0e620ffaaaca4908b49" exitCode=0 Oct 29 23:37:01.928101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:01.928030 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" event={"ID":"97427e0e-39e7-4ff2-926e-2357a1ac7b05","Type":"ContainerDied","Data":"12d3dc22b53907cf8e7e89978243599445203854fac3a0e620ffaaaca4908b49"} Oct 29 23:37:02.150817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.150787 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:37:02.192870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.192837 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-xks-ca\") pod \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " Oct 29 23:37:02.193080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.192929 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-kube-api-access-g6pkf\" (UniqueName: \"kubernetes.io/projected/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-kube-api-access-g6pkf\") pod \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " Oct 29 23:37:02.193945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.193912 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-openshift-ca\") pod \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\" (UID: \"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64\") " Oct 29 23:37:02.194574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.194546 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-openshift-ca" pod "f1ee1d8d-96b9-4fe2-9371-dac4613d4f64" (UID: "f1ee1d8d-96b9-4fe2-9371-dac4613d4f64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:02.195519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.195496 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-xks-ca" pod "f1ee1d8d-96b9-4fe2-9371-dac4613d4f64" (UID: "f1ee1d8d-96b9-4fe2-9371-dac4613d4f64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:37:02.195948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.195927 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-kube-api-access-g6pkf" pod "f1ee1d8d-96b9-4fe2-9371-dac4613d4f64" (UID: "f1ee1d8d-96b9-4fe2-9371-dac4613d4f64"). InnerVolumeSpecName "kube-api-access-g6pkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:02.296147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.296097 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:02.296147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.296141 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:02.296147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.296151 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6pkf\" (UniqueName: \"kubernetes.io/projected/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64-kube-api-access-g6pkf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:02.939073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.939040 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" Oct 29 23:37:02.939466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.939069 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt" event={"ID":"f1ee1d8d-96b9-4fe2-9371-dac4613d4f64","Type":"ContainerDied","Data":"41bae9dce33c0f5c6ae7d24cb14588d3f6b7a8e4b1c663e97a064753a0fff018"} Oct 29 23:37:02.939466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:02.939138 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41bae9dce33c0f5c6ae7d24cb14588d3f6b7a8e4b1c663e97a064753a0fff018" Oct 29 23:37:03.143817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.143787 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:37:03.211090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.211054 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97427e0e-39e7-4ff2-926e-2357a1ac7b05-kube-api-access-xmj2f\" (UniqueName: \"kubernetes.io/projected/97427e0e-39e7-4ff2-926e-2357a1ac7b05-kube-api-access-xmj2f\") pod \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " Oct 29 23:37:03.211295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.211279 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-xks-ca\") pod \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " Oct 29 23:37:03.211375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.211332 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-openshift-ca\") pod \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\" (UID: \"97427e0e-39e7-4ff2-926e-2357a1ac7b05\") " Oct 29 23:37:03.211889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.211855 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-openshift-ca" pod "97427e0e-39e7-4ff2-926e-2357a1ac7b05" (UID: "97427e0e-39e7-4ff2-926e-2357a1ac7b05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:03.212102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.212083 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:03.213557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.213535 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-xks-ca" pod "97427e0e-39e7-4ff2-926e-2357a1ac7b05" (UID: "97427e0e-39e7-4ff2-926e-2357a1ac7b05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:37:03.213847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.213826 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97427e0e-39e7-4ff2-926e-2357a1ac7b05-kube-api-access-xmj2f" pod "97427e0e-39e7-4ff2-926e-2357a1ac7b05" (UID: "97427e0e-39e7-4ff2-926e-2357a1ac7b05"). InnerVolumeSpecName "kube-api-access-xmj2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:03.314202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.314164 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmj2f\" (UniqueName: \"kubernetes.io/projected/97427e0e-39e7-4ff2-926e-2357a1ac7b05-kube-api-access-xmj2f\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:03.314202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.314200 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97427e0e-39e7-4ff2-926e-2357a1ac7b05-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:03.950191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.950159 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" Oct 29 23:37:03.950716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.950197 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6" event={"ID":"97427e0e-39e7-4ff2-926e-2357a1ac7b05","Type":"ContainerDied","Data":"ccfe9c4c969d34973027d126da18edffb5f1c2c3ccd2fbb0a5b1e9a3f4667e2d"} Oct 29 23:37:03.950716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:03.950253 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccfe9c4c969d34973027d126da18edffb5f1c2c3ccd2fbb0a5b1e9a3f4667e2d" Oct 29 23:37:09.242968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.242933 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r"] Oct 29 23:37:09.243752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.243736 5736 cpu_manager.go:414] "containerMap: removing container" podUID="107c1179-356a-4afb-9fe5-6b0dad4a2868" containerName="revoke-keys" Oct 29 23:37:09.243815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.243753 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107c1179-356a-4afb-9fe5-6b0dad4a2868" containerName="revoke-keys" Oct 29 23:37:09.243815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.243783 5736 cpu_manager.go:414] "containerMap: removing container" podUID="97427e0e-39e7-4ff2-926e-2357a1ac7b05" containerName="revoke-keys" Oct 29 23:37:09.243815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.243789 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97427e0e-39e7-4ff2-926e-2357a1ac7b05" containerName="revoke-keys" Oct 29 23:37:09.243815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.243813 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f1ee1d8d-96b9-4fe2-9371-dac4613d4f64" containerName="revoke-keys" Oct 29 23:37:09.243953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.243820 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ee1d8d-96b9-4fe2-9371-dac4613d4f64" containerName="revoke-keys" Oct 29 23:37:09.243986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.243966 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="97427e0e-39e7-4ff2-926e-2357a1ac7b05" containerName="revoke-keys" Oct 29 23:37:09.243986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.243982 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ee1d8d-96b9-4fe2-9371-dac4613d4f64" containerName="revoke-keys" Oct 29 23:37:09.244054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.243994 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="107c1179-356a-4afb-9fe5-6b0dad4a2868" containerName="revoke-keys" Oct 29 23:37:09.249797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.249763 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.270367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.270316 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r"] Oct 29 23:37:09.276107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.276074 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:37:09.276107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.276076 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:37:09.419872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.419841 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3f875a1-qbp6r\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.420085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.420054 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdf6\" (UniqueName: \"kubernetes.io/projected/dcd3d048-8f0e-4217-98b4-f7400ab490f7-kube-api-access-pmdf6\") pod \"maas-api-revoke-sub-f3f875a1-qbp6r\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.420139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.420103 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3f875a1-qbp6r\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.522293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.522207 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdf6\" (UniqueName: \"kubernetes.io/projected/dcd3d048-8f0e-4217-98b4-f7400ab490f7-kube-api-access-pmdf6\") pod \"maas-api-revoke-sub-f3f875a1-qbp6r\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.522293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.522269 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3f875a1-qbp6r\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.522511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.522474 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3f875a1-qbp6r\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.523216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.523179 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3f875a1-qbp6r\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.524762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.524735 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3f875a1-qbp6r\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.549129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.549101 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdf6\" (UniqueName: \"kubernetes.io/projected/dcd3d048-8f0e-4217-98b4-f7400ab490f7-kube-api-access-pmdf6\") pod \"maas-api-revoke-sub-f3f875a1-qbp6r\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.578826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.578801 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:09.635902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.635873 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f"] Oct 29 23:37:09.908904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:09.908870 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r"] Oct 29 23:37:09.911113 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:37:09.911083 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd3d048_8f0e_4217_98b4_f7400ab490f7.slice/crio-162acd319f669942b612d0f6a71ba8b27520de51e20c398269484e971ac41cdd WatchSource:0}: Error finding container 162acd319f669942b612d0f6a71ba8b27520de51e20c398269484e971ac41cdd: Status 404 returned error can't find the container with id 162acd319f669942b612d0f6a71ba8b27520de51e20c398269484e971ac41cdd Oct 29 23:37:10.071606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:10.049360 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" event={"ID":"dcd3d048-8f0e-4217-98b4-f7400ab490f7","Type":"ContainerStarted","Data":"162acd319f669942b612d0f6a71ba8b27520de51e20c398269484e971ac41cdd"} Oct 29 23:37:10.458343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:10.458311 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6"] Oct 29 23:37:11.074609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:11.070156 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" event={"ID":"dcd3d048-8f0e-4217-98b4-f7400ab490f7","Type":"ContainerStarted","Data":"b73d9e554077fc3fe54169e1db0e3851dabb7664f45ca5d5c6753958772ae007"} Oct 29 23:37:11.115939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:11.115876 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" podStartSLOduration=2.115858772 podStartE2EDuration="2.115858772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:11.113939946 +0000 UTC m=+3228.229859546" watchObservedRunningTime="2026-10-29 23:37:11.115858772 +0000 UTC m=+3228.231778381" Oct 29 23:37:12.106051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:12.106011 5736 generic.go:356] "Generic (PLEG): container finished" podID="dcd3d048-8f0e-4217-98b4-f7400ab490f7" containerID="b73d9e554077fc3fe54169e1db0e3851dabb7664f45ca5d5c6753958772ae007" exitCode=0 Oct 29 23:37:12.106501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:12.106107 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" event={"ID":"dcd3d048-8f0e-4217-98b4-f7400ab490f7","Type":"ContainerDied","Data":"b73d9e554077fc3fe54169e1db0e3851dabb7664f45ca5d5c6753958772ae007"} Oct 29 23:37:13.281464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.281435 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:13.432874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.432793 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd3d048-8f0e-4217-98b4-f7400ab490f7-kube-api-access-pmdf6\" (UniqueName: \"kubernetes.io/projected/dcd3d048-8f0e-4217-98b4-f7400ab490f7-kube-api-access-pmdf6\") pod \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " Oct 29 23:37:13.433042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.432965 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-openshift-ca\") pod \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " Oct 29 23:37:13.433042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.433002 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-xks-ca\") pod \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\" (UID: \"dcd3d048-8f0e-4217-98b4-f7400ab490f7\") " Oct 29 23:37:13.433348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.433321 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-openshift-ca" pod "dcd3d048-8f0e-4217-98b4-f7400ab490f7" (UID: "dcd3d048-8f0e-4217-98b4-f7400ab490f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:13.433811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.433791 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:13.434970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.434944 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd3d048-8f0e-4217-98b4-f7400ab490f7-kube-api-access-pmdf6" pod "dcd3d048-8f0e-4217-98b4-f7400ab490f7" (UID: "dcd3d048-8f0e-4217-98b4-f7400ab490f7"). InnerVolumeSpecName "kube-api-access-pmdf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:13.435160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.435147 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-xks-ca" pod "dcd3d048-8f0e-4217-98b4-f7400ab490f7" (UID: "dcd3d048-8f0e-4217-98b4-f7400ab490f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:37:13.535330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.535291 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd3d048-8f0e-4217-98b4-f7400ab490f7-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:13.535330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:13.535324 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmdf6\" (UniqueName: \"kubernetes.io/projected/dcd3d048-8f0e-4217-98b4-f7400ab490f7-kube-api-access-pmdf6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:14.124251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.124224 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" Oct 29 23:37:14.124251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.124243 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r" event={"ID":"dcd3d048-8f0e-4217-98b4-f7400ab490f7","Type":"ContainerDied","Data":"162acd319f669942b612d0f6a71ba8b27520de51e20c398269484e971ac41cdd"} Oct 29 23:37:14.124476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.124291 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="162acd319f669942b612d0f6a71ba8b27520de51e20c398269484e971ac41cdd" Oct 29 23:37:14.636372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.636332 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" podUID="075bdf77-14ec-417e-bf3c-1405e3970272" containerName="payload-processing" containerID="cri-o://6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460" gracePeriod=25 Oct 29 23:37:14.929788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.929761 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:37:14.969869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.969843 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-plugins-config-volume\") pod \"075bdf77-14ec-417e-bf3c-1405e3970272\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " Oct 29 23:37:14.970017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.969878 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-kube-api-access-7hwpw\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-kube-api-access-7hwpw\") pod \"075bdf77-14ec-417e-bf3c-1405e3970272\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " Oct 29 23:37:14.970068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.970020 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-routing-overlay-volume\") pod \"075bdf77-14ec-417e-bf3c-1405e3970272\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " Oct 29 23:37:14.970193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.970177 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-provider-credentials\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-provider-credentials\") pod \"075bdf77-14ec-417e-bf3c-1405e3970272\" (UID: \"075bdf77-14ec-417e-bf3c-1405e3970272\") " Oct 29 23:37:14.970408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.970377 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-plugins-config-volume" pod "075bdf77-14ec-417e-bf3c-1405e3970272" (UID: "075bdf77-14ec-417e-bf3c-1405e3970272"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:14.970528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.970480 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-routing-overlay-volume" pod "075bdf77-14ec-417e-bf3c-1405e3970272" (UID: "075bdf77-14ec-417e-bf3c-1405e3970272"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:14.971085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.971058 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:14.971236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.971089 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075bdf77-14ec-417e-bf3c-1405e3970272-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:14.976340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.972705 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-kube-api-access-7hwpw" pod "075bdf77-14ec-417e-bf3c-1405e3970272" (UID: "075bdf77-14ec-417e-bf3c-1405e3970272"). InnerVolumeSpecName "kube-api-access-7hwpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:14.976340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:14.973364 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-provider-credentials" pod "075bdf77-14ec-417e-bf3c-1405e3970272" (UID: "075bdf77-14ec-417e-bf3c-1405e3970272"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:15.073622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.073575 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:15.073622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.073626 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hwpw\" (UniqueName: \"kubernetes.io/projected/075bdf77-14ec-417e-bf3c-1405e3970272-kube-api-access-7hwpw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:15.134423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.134394 5736 generic.go:356] "Generic (PLEG): container finished" podID="075bdf77-14ec-417e-bf3c-1405e3970272" containerID="6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460" exitCode=0 Oct 29 23:37:15.134545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.134440 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" Oct 29 23:37:15.134545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.134445 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" event={"ID":"075bdf77-14ec-417e-bf3c-1405e3970272","Type":"ContainerDied","Data":"6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460"} Oct 29 23:37:15.134639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.134571 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f" event={"ID":"075bdf77-14ec-417e-bf3c-1405e3970272","Type":"ContainerDied","Data":"4713cb4e47ac58caa957736e0f8b5f61d6d0d14773b8d5eb113c599703859096"} Oct 29 23:37:15.134639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.134635 5736 scope.go:122] "RemoveContainer" containerID="6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460" Oct 29 23:37:15.145875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.145853 5736 scope.go:122] "RemoveContainer" containerID="6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460" Oct 29 23:37:15.146175 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:15.146152 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460\": container with ID starting with 6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460 not found: ID does not exist" containerID="6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460" Oct 29 23:37:15.146223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.146183 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460"} err="failed to get container status \"6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460\": rpc error: code = NotFound desc = could not find container \"6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460\": container with ID starting with 6d68e692ec80f7a7a525df17d99a7a3933fd30ff7fa0f895ee14262bfc0bb460 not found: ID does not exist" Oct 29 23:37:15.188908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.188882 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f"] Oct 29 23:37:15.193850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.193828 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6f162a/payload-processing-external-model-e2e-extmod-a-6f162a-79cctfn9f"] Oct 29 23:37:15.458717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.458621 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" podUID="3cfdcacc-5638-4252-9100-5a9ebe006df5" containerName="payload-processing" containerID="cri-o://97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe" gracePeriod=25 Oct 29 23:37:15.808964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.808934 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:37:15.992411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.992378 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-plugins-config-volume\") pod \"3cfdcacc-5638-4252-9100-5a9ebe006df5\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " Oct 29 23:37:15.992629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.992617 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-provider-credentials\") pod \"3cfdcacc-5638-4252-9100-5a9ebe006df5\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " Oct 29 23:37:15.992698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.992666 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-kube-api-access-rvmnr\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-kube-api-access-rvmnr\") pod \"3cfdcacc-5638-4252-9100-5a9ebe006df5\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " Oct 29 23:37:15.992807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.992786 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-routing-overlay-volume\") pod \"3cfdcacc-5638-4252-9100-5a9ebe006df5\" (UID: \"3cfdcacc-5638-4252-9100-5a9ebe006df5\") " Oct 29 23:37:15.992956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.992795 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-plugins-config-volume" pod "3cfdcacc-5638-4252-9100-5a9ebe006df5" (UID: "3cfdcacc-5638-4252-9100-5a9ebe006df5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:15.993188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.993153 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-routing-overlay-volume" pod "3cfdcacc-5638-4252-9100-5a9ebe006df5" (UID: "3cfdcacc-5638-4252-9100-5a9ebe006df5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:15.993877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.993846 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:15.993983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.993878 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfdcacc-5638-4252-9100-5a9ebe006df5-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:15.995083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.995056 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-provider-credentials" pod "3cfdcacc-5638-4252-9100-5a9ebe006df5" (UID: "3cfdcacc-5638-4252-9100-5a9ebe006df5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:15.995192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:15.995171 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-kube-api-access-rvmnr" pod "3cfdcacc-5638-4252-9100-5a9ebe006df5" (UID: "3cfdcacc-5638-4252-9100-5a9ebe006df5"). InnerVolumeSpecName "kube-api-access-rvmnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:16.095649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.095545 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:16.095649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.095575 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvmnr\" (UniqueName: \"kubernetes.io/projected/3cfdcacc-5638-4252-9100-5a9ebe006df5-kube-api-access-rvmnr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:16.146422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.146389 5736 generic.go:356] "Generic (PLEG): container finished" podID="3cfdcacc-5638-4252-9100-5a9ebe006df5" containerID="97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe" exitCode=0 Oct 29 23:37:16.146696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.146457 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" Oct 29 23:37:16.146696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.146479 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" event={"ID":"3cfdcacc-5638-4252-9100-5a9ebe006df5","Type":"ContainerDied","Data":"97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe"} Oct 29 23:37:16.146696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.146542 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6" event={"ID":"3cfdcacc-5638-4252-9100-5a9ebe006df5","Type":"ContainerDied","Data":"c04daa221c818dbcc8f6c8b98251bd7562dfca0c9ade3e6e695dac83c02d2def"} Oct 29 23:37:16.146696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.146562 5736 scope.go:122] "RemoveContainer" containerID="97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe" Oct 29 23:37:16.160550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.160532 5736 scope.go:122] "RemoveContainer" containerID="97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe" Oct 29 23:37:16.160895 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:16.160840 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe\": container with ID starting with 97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe not found: ID does not exist" containerID="97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe" Oct 29 23:37:16.160895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.160874 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe"} err="failed to get container status \"97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe\": rpc error: code = NotFound desc = could not find container \"97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe\": container with ID starting with 97a2959edcacdd1d84c419405936f716e056b69d11823606e1852280565089fe not found: ID does not exist" Oct 29 23:37:16.216217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.216173 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6"] Oct 29 23:37:16.220494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:16.220463 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb367d/payload-processing-external-model-e2e-extmod-b-fb367d-75c856zp6"] Oct 29 23:37:17.008104 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.008069 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075bdf77-14ec-417e-bf3c-1405e3970272" path="/var/lib/kubelet/pods/075bdf77-14ec-417e-bf3c-1405e3970272/volumes" Oct 29 23:37:17.008684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.008666 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cfdcacc-5638-4252-9100-5a9ebe006df5" path="/var/lib/kubelet/pods/3cfdcacc-5638-4252-9100-5a9ebe006df5/volumes" Oct 29 23:37:17.009119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.009089 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr"] Oct 29 23:37:17.009945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.009917 5736 cpu_manager.go:414] "containerMap: removing container" podUID="dcd3d048-8f0e-4217-98b4-f7400ab490f7" containerName="revoke-keys" Oct 29 23:37:17.009945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.009940 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd3d048-8f0e-4217-98b4-f7400ab490f7" containerName="revoke-keys" Oct 29 23:37:17.010138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.009987 5736 cpu_manager.go:414] "containerMap: removing container" podUID="075bdf77-14ec-417e-bf3c-1405e3970272" containerName="payload-processing" Oct 29 23:37:17.010138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.009997 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075bdf77-14ec-417e-bf3c-1405e3970272" containerName="payload-processing" Oct 29 23:37:17.010138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.010048 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3cfdcacc-5638-4252-9100-5a9ebe006df5" containerName="payload-processing" Oct 29 23:37:17.010138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.010056 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfdcacc-5638-4252-9100-5a9ebe006df5" containerName="payload-processing" Oct 29 23:37:17.010348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.010290 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd3d048-8f0e-4217-98b4-f7400ab490f7" containerName="revoke-keys" Oct 29 23:37:17.010348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.010312 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="075bdf77-14ec-417e-bf3c-1405e3970272" containerName="payload-processing" Oct 29 23:37:17.010348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.010329 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfdcacc-5638-4252-9100-5a9ebe006df5" containerName="payload-processing" Oct 29 23:37:17.016029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.016001 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.031050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.031021 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6d9e36\"/\"kube-root-ca.crt\"" Oct 29 23:37:17.031314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.031288 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6d9e36\"/\"test-model-6d9e36-kserve-self-signed-certs\"" Oct 29 23:37:17.033287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.033261 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr"] Oct 29 23:37:17.043142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.043114 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6d9e36\"/\"openshift-service-ca.crt\"" Oct 29 23:37:17.043142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.043121 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6d9e36\"/\"default-dockercfg-t6nvf\"" Oct 29 23:37:17.050045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.050020 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v"] Oct 29 23:37:17.056797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.056775 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6211d41d-vjc4v"] Oct 29 23:37:17.112254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.112210 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-tmp-dir\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.112440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.112324 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-model-cache\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.112440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.112374 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-home\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.112571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.112458 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55389672-b516-4422-b514-3ca1355c361a-tls-certs\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.112571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.112515 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-dshm\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.112764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.112627 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2nrf\" (UniqueName: \"kubernetes.io/projected/55389672-b516-4422-b514-3ca1355c361a-kube-api-access-t2nrf\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.214356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.214309 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55389672-b516-4422-b514-3ca1355c361a-tls-certs\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.214571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.214414 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-dshm\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.214571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.214510 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2nrf\" (UniqueName: \"kubernetes.io/projected/55389672-b516-4422-b514-3ca1355c361a-kube-api-access-t2nrf\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.214777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.214735 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-tmp-dir\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.214853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.214832 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-model-cache\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.214911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.214865 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-home\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.215180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.215150 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-model-cache\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.215180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.215170 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-tmp-dir\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.215320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.215241 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-home\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.216929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.216901 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-dshm\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.217297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.217279 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55389672-b516-4422-b514-3ca1355c361a-tls-certs\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.238690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.238657 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2nrf\" (UniqueName: \"kubernetes.io/projected/55389672-b516-4422-b514-3ca1355c361a-kube-api-access-t2nrf\") pod \"test-model-6d9e36-kserve-84dc5d565c-g9mjr\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.338655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.338183 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:17.835006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:17.834973 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr"] Oct 29 23:37:17.835997 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:37:17.835960 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55389672_b516_4422_b514_3ca1355c361a.slice/crio-95445d33f2831cf9d76a1542bb577d2fc750875e0c110b56d2f5b716d4e30b7b WatchSource:0}: Error finding container 95445d33f2831cf9d76a1542bb577d2fc750875e0c110b56d2f5b716d4e30b7b: Status 404 returned error can't find the container with id 95445d33f2831cf9d76a1542bb577d2fc750875e0c110b56d2f5b716d4e30b7b Oct 29 23:37:18.170573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:18.170486 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" event={"ID":"55389672-b516-4422-b514-3ca1355c361a","Type":"ContainerStarted","Data":"7068750864e51e7939aa7b83ce60996ed0f9ce402437b4a0b9b2e271a07bd9e3"} Oct 29 23:37:18.170573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:18.170535 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" event={"ID":"55389672-b516-4422-b514-3ca1355c361a","Type":"ContainerStarted","Data":"95445d33f2831cf9d76a1542bb577d2fc750875e0c110b56d2f5b716d4e30b7b"} Oct 29 23:37:18.214589 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:18.214529 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" podStartSLOduration=2.21451487 podStartE2EDuration="2.21451487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:18.210756264 +0000 UTC m=+3235.326675870" watchObservedRunningTime="2026-10-29 23:37:18.21451487 +0000 UTC m=+3235.330434525" Oct 29 23:37:18.348487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:18.348448 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:19.007782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:19.007746 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c2012a-8e35-439d-bf4b-a610ea623c78" path="/var/lib/kubelet/pods/70c2012a-8e35-439d-bf4b-a610ea623c78/volumes" Oct 29 23:37:20.323891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:20.323853 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx"] Oct 29 23:37:20.324371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:20.324061 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" podUID="7292011e-66f4-46ee-b0b0-6b023e570dbe" containerName="sim" containerID="cri-o://e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f" gracePeriod=30 Oct 29 23:37:20.673882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:20.673842 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:37:20.797315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:20.797276 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7292011e-66f4-46ee-b0b0-6b023e570dbe-kube-api-access-mh5b6\" (UniqueName: \"kubernetes.io/projected/7292011e-66f4-46ee-b0b0-6b023e570dbe-kube-api-access-mh5b6\") pod \"7292011e-66f4-46ee-b0b0-6b023e570dbe\" (UID: \"7292011e-66f4-46ee-b0b0-6b023e570dbe\") " Oct 29 23:37:20.797648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:20.797619 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7292011e-66f4-46ee-b0b0-6b023e570dbe-tls\" (UniqueName: \"kubernetes.io/secret/7292011e-66f4-46ee-b0b0-6b023e570dbe-tls\") pod \"7292011e-66f4-46ee-b0b0-6b023e570dbe\" (UID: \"7292011e-66f4-46ee-b0b0-6b023e570dbe\") " Oct 29 23:37:20.800738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:20.800692 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7292011e-66f4-46ee-b0b0-6b023e570dbe-kube-api-access-mh5b6" pod "7292011e-66f4-46ee-b0b0-6b023e570dbe" (UID: "7292011e-66f4-46ee-b0b0-6b023e570dbe"). InnerVolumeSpecName "kube-api-access-mh5b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:20.800962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:20.800942 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7292011e-66f4-46ee-b0b0-6b023e570dbe-tls" pod "7292011e-66f4-46ee-b0b0-6b023e570dbe" (UID: "7292011e-66f4-46ee-b0b0-6b023e570dbe"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:37:20.900453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:20.900380 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh5b6\" (UniqueName: \"kubernetes.io/projected/7292011e-66f4-46ee-b0b0-6b023e570dbe-kube-api-access-mh5b6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:20.900453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:20.900420 5736 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7292011e-66f4-46ee-b0b0-6b023e570dbe-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:21.247662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.247540 5736 generic.go:356] "Generic (PLEG): container finished" podID="7292011e-66f4-46ee-b0b0-6b023e570dbe" containerID="e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f" exitCode=0 Oct 29 23:37:21.247662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.247634 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" event={"ID":"7292011e-66f4-46ee-b0b0-6b023e570dbe","Type":"ContainerDied","Data":"e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f"} Oct 29 23:37:21.247947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.247688 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" event={"ID":"7292011e-66f4-46ee-b0b0-6b023e570dbe","Type":"ContainerDied","Data":"a649d9ad90bbacde6f60a6191b75036e108e6149b5c1c2bda39544b32f324b2c"} Oct 29 23:37:21.247947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.247714 5736 scope.go:122] "RemoveContainer" containerID="e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f" Oct 29 23:37:21.247947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.247942 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx" Oct 29 23:37:21.264944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.264917 5736 scope.go:122] "RemoveContainer" containerID="e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f" Oct 29 23:37:21.265294 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:21.265270 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f\": container with ID starting with e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f not found: ID does not exist" containerID="e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f" Oct 29 23:37:21.265365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.265302 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f"} err="failed to get container status \"e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f\": rpc error: code = NotFound desc = could not find container \"e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f\": container with ID starting with e85a38010043d878c6cb4ce59cc258adcbf375cf4c74babf1ed6d6a17d3f663f not found: ID does not exist" Oct 29 23:37:21.304333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.304291 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx"] Oct 29 23:37:21.313943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.313911 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-eba670/e2e-usage-sim-18d73f-95c45d9f-9fmcx"] Oct 29 23:37:21.574615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.571222 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m"] Oct 29 23:37:21.592604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.590791 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7292011e-66f4-46ee-b0b0-6b023e570dbe" containerName="sim" Oct 29 23:37:21.592604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.590824 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7292011e-66f4-46ee-b0b0-6b023e570dbe" containerName="sim" Oct 29 23:37:21.592604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.591080 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7292011e-66f4-46ee-b0b0-6b023e570dbe" containerName="sim" Oct 29 23:37:21.662606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.623788 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m"] Oct 29 23:37:21.662606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.623913 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.666644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.662927 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:37:21.666644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.663619 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:37:21.738110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.715747 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e256980-chf8m\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.738110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.715878 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e256980-chf8m\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.738110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.716035 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzhjq\" (UniqueName: \"kubernetes.io/projected/58937dda-a893-4379-bf97-4efc3c4d3529-kube-api-access-fzhjq\") pod \"maas-api-revoke-sub-5e256980-chf8m\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.758339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.756796 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4"] Oct 29 23:37:21.769373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.767175 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:21.787010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.785301 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"payload-processing-external-model-e2e--addccaec-dockercfg-2jmn5\"" Oct 29 23:37:21.788242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.787695 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"agc-cred-0dd886d5c29d\"" Oct 29 23:37:21.788242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.787951 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"routing-overlay\"" Oct 29 23:37:21.788242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.788046 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"openshift-service-ca.crt\"" Oct 29 23:37:21.788242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.788152 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"kube-root-ca.crt\"" Oct 29 23:37:21.788242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.788206 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e1769a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e1769a\"" Oct 29 23:37:21.820029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.818086 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzhjq\" (UniqueName: \"kubernetes.io/projected/58937dda-a893-4379-bf97-4efc3c4d3529-kube-api-access-fzhjq\") pod \"maas-api-revoke-sub-5e256980-chf8m\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.820029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.818275 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e256980-chf8m\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.820029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.818353 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e256980-chf8m\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.820365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.820091 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e256980-chf8m\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.829636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.821881 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e256980-chf8m\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.829636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.825293 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4"] Oct 29 23:37:21.858600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.852202 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzhjq\" (UniqueName: \"kubernetes.io/projected/58937dda-a893-4379-bf97-4efc3c4d3529-kube-api-access-fzhjq\") pod \"maas-api-revoke-sub-5e256980-chf8m\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:21.919527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.919483 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:21.919808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.919773 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5tm\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-kube-api-access-xg5tm\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:21.919950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.919884 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:21.920237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:21.920214 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.023606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.022961 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.023606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.023170 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5tm\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-kube-api-access-xg5tm\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.023606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.023220 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.023606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.023334 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.048613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.033575 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.048613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.037472 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.075621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.075128 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.094655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.075943 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:22.094655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.083300 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5tm\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-kube-api-access-xg5tm\") pod \"payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.114256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.107098 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:22.677636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.672316 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4"] Oct 29 23:37:22.708749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:22.708719 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m"] Oct 29 23:37:22.710121 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:37:22.710092 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58937dda_a893_4379_bf97_4efc3c4d3529.slice/crio-d0cc311ac1251ec5e7e4a22d06a0c2a026b5b2e526f6677f75840af0f18e914f WatchSource:0}: Error finding container d0cc311ac1251ec5e7e4a22d06a0c2a026b5b2e526f6677f75840af0f18e914f: Status 404 returned error can't find the container with id d0cc311ac1251ec5e7e4a22d06a0c2a026b5b2e526f6677f75840af0f18e914f Oct 29 23:37:23.011200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.011162 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7292011e-66f4-46ee-b0b0-6b023e570dbe" path="/var/lib/kubelet/pods/7292011e-66f4-46ee-b0b0-6b023e570dbe/volumes" Oct 29 23:37:23.351421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.351023 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf"] Oct 29 23:37:23.361059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.358065 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.371622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.371379 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf"] Oct 29 23:37:23.371622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.371452 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" event={"ID":"765f41c8-ddd4-4813-b475-eb1723bb2091","Type":"ContainerStarted","Data":"afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a"} Oct 29 23:37:23.371622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.371488 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" event={"ID":"765f41c8-ddd4-4813-b475-eb1723bb2091","Type":"ContainerStarted","Data":"5db017de07ea745c770f264c02f643f517dc4f4e0ab647f306ef63fd1ba37a76"} Oct 29 23:37:23.372543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.372517 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:23.375407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.375370 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" event={"ID":"58937dda-a893-4379-bf97-4efc3c4d3529","Type":"ContainerStarted","Data":"fc988303a88d1938804372ae51cdd6f43de597a646100e9bdc696a3f608af5ff"} Oct 29 23:37:23.375539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.375420 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" event={"ID":"58937dda-a893-4379-bf97-4efc3c4d3529","Type":"ContainerStarted","Data":"d0cc311ac1251ec5e7e4a22d06a0c2a026b5b2e526f6677f75840af0f18e914f"} Oct 29 23:37:23.375646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.375624 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"payload-processing-external-model-e2e--86ce1f4b-dockercfg-jxqhv\"" Oct 29 23:37:23.376486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.376210 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"kube-root-ca.crt\"" Oct 29 23:37:23.376486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.376308 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"routing-overlay\"" Oct 29 23:37:23.376486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.376383 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"agc-cred-ebd330ca5bd8\"" Oct 29 23:37:23.376486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.376424 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"openshift-service-ca.crt\"" Oct 29 23:37:23.376486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.376459 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-45036b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-45036b\"" Oct 29 23:37:23.443192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.443116 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" podStartSLOduration=2.44309779 podStartE2EDuration="2.44309779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:23.441883809 +0000 UTC m=+3240.557803414" watchObservedRunningTime="2026-10-29 23:37:23.44309779 +0000 UTC m=+3240.559017396" Oct 29 23:37:23.483360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.483249 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" podStartSLOduration=2.483236521 podStartE2EDuration="2.483236521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:23.482412731 +0000 UTC m=+3240.598332506" watchObservedRunningTime="2026-10-29 23:37:23.483236521 +0000 UTC m=+3240.599156127" Oct 29 23:37:23.504441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.504404 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpd2b\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-kube-api-access-zpd2b\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.504575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.504496 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.504657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.504571 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.504935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.504902 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.606568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.606535 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpd2b\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-kube-api-access-zpd2b\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.606742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.606589 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.606742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.606732 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.606969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.606947 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.607100 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:23.607080 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-45036b/agc-cred-ebd330ca5bd8: secret "agc-cred-ebd330ca5bd8" not found Oct 29 23:37:23.607173 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:23.607102 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf: secret "agc-cred-ebd330ca5bd8" not found Oct 29 23:37:23.607173 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:23.607163 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials podName:da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb nodeName:}" failed. No retries permitted until 2026-10-29 23:37:24.107144781 +0000 UTC m=+3241.223064367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" (UID: "da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb") : secret "agc-cred-ebd330ca5bd8" not found Oct 29 23:37:23.607362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.607341 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.607688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.607665 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:23.630684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:23.630652 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpd2b\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-kube-api-access-zpd2b\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:24.115906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:24.115869 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:24.116335 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:24.116013 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-45036b/agc-cred-ebd330ca5bd8: secret "agc-cred-ebd330ca5bd8" not found Oct 29 23:37:24.116335 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:24.116033 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf: secret "agc-cred-ebd330ca5bd8" not found Oct 29 23:37:24.116335 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:24.116085 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials podName:da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb nodeName:}" failed. No retries permitted until 2026-10-29 23:37:25.116070612 +0000 UTC m=+3242.231990196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" (UID: "da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb") : secret "agc-cred-ebd330ca5bd8" not found Oct 29 23:37:24.387034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:24.386995 5736 generic.go:356] "Generic (PLEG): container finished" podID="58937dda-a893-4379-bf97-4efc3c4d3529" containerID="fc988303a88d1938804372ae51cdd6f43de597a646100e9bdc696a3f608af5ff" exitCode=0 Oct 29 23:37:24.387236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:24.387080 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" event={"ID":"58937dda-a893-4379-bf97-4efc3c4d3529","Type":"ContainerDied","Data":"fc988303a88d1938804372ae51cdd6f43de597a646100e9bdc696a3f608af5ff"} Oct 29 23:37:25.131271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.131225 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:25.131712 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:25.131287 5736 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-45036b/agc-cred-ebd330ca5bd8: secret "agc-cred-ebd330ca5bd8" not found Oct 29 23:37:25.131712 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:25.131308 5736 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf: secret "agc-cred-ebd330ca5bd8" not found Oct 29 23:37:25.131712 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:25.131356 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials podName:da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb nodeName:}" failed. No retries permitted until 2026-10-29 23:37:27.131342559 +0000 UTC m=+3244.247262156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" (UID: "da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb") : secret "agc-cred-ebd330ca5bd8" not found Oct 29 23:37:25.299902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.299863 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf"] Oct 29 23:37:25.300569 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:25.300531 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" podUID="da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb" Oct 29 23:37:25.396240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.396143 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:25.413668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.413638 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:25.542131 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.542094 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-plugins-config-volume\") pod \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " Oct 29 23:37:25.542310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.542291 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-kube-api-access-zpd2b\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-kube-api-access-zpd2b\") pod \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " Oct 29 23:37:25.542524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.542492 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-plugins-config-volume" pod "da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb" (UID: "da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:25.542524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.542513 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-routing-overlay-volume\") pod \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\" (UID: \"da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb\") " Oct 29 23:37:25.542893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.542873 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-routing-overlay-volume" pod "da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb" (UID: "da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:25.543774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.543745 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:25.543897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.543777 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:25.544846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.544818 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-kube-api-access-zpd2b" pod "da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb" (UID: "da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb"). InnerVolumeSpecName "kube-api-access-zpd2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:25.583449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.583424 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:25.680656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.651126 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpd2b\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-kube-api-access-zpd2b\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:25.766896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.765223 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-openshift-ca\") pod \"58937dda-a893-4379-bf97-4efc3c4d3529\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " Oct 29 23:37:25.766896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.765280 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58937dda-a893-4379-bf97-4efc3c4d3529-kube-api-access-fzhjq\" (UniqueName: \"kubernetes.io/projected/58937dda-a893-4379-bf97-4efc3c4d3529-kube-api-access-fzhjq\") pod \"58937dda-a893-4379-bf97-4efc3c4d3529\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " Oct 29 23:37:25.766896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.765722 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-xks-ca\") pod \"58937dda-a893-4379-bf97-4efc3c4d3529\" (UID: \"58937dda-a893-4379-bf97-4efc3c4d3529\") " Oct 29 23:37:25.782843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.782799 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-openshift-ca" pod "58937dda-a893-4379-bf97-4efc3c4d3529" (UID: "58937dda-a893-4379-bf97-4efc3c4d3529"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:25.859842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.839859 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58937dda-a893-4379-bf97-4efc3c4d3529-kube-api-access-fzhjq" pod "58937dda-a893-4379-bf97-4efc3c4d3529" (UID: "58937dda-a893-4379-bf97-4efc3c4d3529"). InnerVolumeSpecName "kube-api-access-fzhjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:25.917115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.917050 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:25.917115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.917087 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzhjq\" (UniqueName: \"kubernetes.io/projected/58937dda-a893-4379-bf97-4efc3c4d3529-kube-api-access-fzhjq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:25.924642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:25.921412 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-xks-ca" pod "58937dda-a893-4379-bf97-4efc3c4d3529" (UID: "58937dda-a893-4379-bf97-4efc3c4d3529"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:37:26.019208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:26.019175 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58937dda-a893-4379-bf97-4efc3c4d3529-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:26.410227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:26.408812 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" Oct 29 23:37:26.410227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:26.408836 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf" Oct 29 23:37:26.410227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:26.408866 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m" event={"ID":"58937dda-a893-4379-bf97-4efc3c4d3529","Type":"ContainerDied","Data":"d0cc311ac1251ec5e7e4a22d06a0c2a026b5b2e526f6677f75840af0f18e914f"} Oct 29 23:37:26.410227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:26.408906 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0cc311ac1251ec5e7e4a22d06a0c2a026b5b2e526f6677f75840af0f18e914f" Oct 29 23:37:26.492635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:26.492557 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf"] Oct 29 23:37:26.497770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:26.497744 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-45036b/payload-processing-external-model-e2e-extmod-a-45036b-8496xmvpf"] Oct 29 23:37:26.526118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:26.526089 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:26.826198 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:26.826161 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:37:27.006480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:27.006439 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb" path="/var/lib/kubelet/pods/da72e0d1-6e15-4872-8c05-3cc9c9b7fcfb/volumes" Oct 29 23:37:27.348540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:27.348502 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:27.370602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:27.369562 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:27.452956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:27.452926 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:37:27.998475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:27.998439 5736 scope.go:122] "RemoveContainer" containerID="afb3b0062d8473490e9a40bb03238e41e704e7a2fa9de20e5238697f71fadd01" Oct 29 23:37:28.351397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.351315 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht"] Oct 29 23:37:28.352562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.352530 5736 cpu_manager.go:414] "containerMap: removing container" podUID="58937dda-a893-4379-bf97-4efc3c4d3529" containerName="revoke-keys" Oct 29 23:37:28.352562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.352560 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58937dda-a893-4379-bf97-4efc3c4d3529" containerName="revoke-keys" Oct 29 23:37:28.352801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.352783 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="58937dda-a893-4379-bf97-4efc3c4d3529" containerName="revoke-keys" Oct 29 23:37:28.359505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.359482 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.378009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.377988 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a8ab3c-openshift-default-dockercfg-2n9mx\"" Oct 29 23:37:28.434291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.434190 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht"] Oct 29 23:37:28.607707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.584114 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-data\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.608411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.608364 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-str5n\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-kube-api-access-str5n\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.619841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.619730 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d529cec-33d9-44f2-b1a6-56dcba642a03-istiod-ca-cert\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.653602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.647788 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-credential-socket\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.653602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.648017 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-socket\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.653602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.648101 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-certs\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.653602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.648172 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-envoy\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.653602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.648378 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-token\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.653602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.648486 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-podinfo\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.769607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.768706 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-podinfo\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.769607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.768797 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-data\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.769607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.768867 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-str5n\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-kube-api-access-str5n\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.769607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.768898 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d529cec-33d9-44f2-b1a6-56dcba642a03-istiod-ca-cert\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.769607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.768981 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-credential-socket\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.769607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.769114 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-socket\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.769607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.769216 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-certs\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.769607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.769263 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-envoy\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.769607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.769440 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-token\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.813733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.773797 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-credential-socket\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.813733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.776954 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-certs\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.813733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.777270 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-socket\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.813733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.779248 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-data\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.813733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.812449 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d529cec-33d9-44f2-b1a6-56dcba642a03-istiod-ca-cert\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.817321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.817291 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-str5n\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-kube-api-access-str5n\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.868647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.864903 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-envoy\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.941602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.940489 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-podinfo\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.945658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.944905 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-token\") pod \"e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:28.986885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:28.984037 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:29.037979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.037934 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl"] Oct 29 23:37:29.042516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.042480 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c76757-lkvfl"] Oct 29 23:37:29.347898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.347868 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht"] Oct 29 23:37:29.350137 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:37:29.350093 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d529cec_33d9_44f2_b1a6_56dcba642a03.slice/crio-1542428581ca2e1d2fe35fe343d842a818afdbc45ed7b0f35fde7b5b55e807f9 WatchSource:0}: Error finding container 1542428581ca2e1d2fe35fe343d842a818afdbc45ed7b0f35fde7b5b55e807f9: Status 404 returned error can't find the container with id 1542428581ca2e1d2fe35fe343d842a818afdbc45ed7b0f35fde7b5b55e807f9 Oct 29 23:37:29.353444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.353407 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:37:29.353562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.353479 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:37:29.353562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.353508 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:37:29.584024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.583981 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" event={"ID":"5d529cec-33d9-44f2-b1a6-56dcba642a03","Type":"ContainerStarted","Data":"d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8"} Oct 29 23:37:29.584200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.584054 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" event={"ID":"5d529cec-33d9-44f2-b1a6-56dcba642a03","Type":"ContainerStarted","Data":"1542428581ca2e1d2fe35fe343d842a818afdbc45ed7b0f35fde7b5b55e807f9"} Oct 29 23:37:29.635768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.635662 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" podStartSLOduration=1.635643304 podStartE2EDuration="1.635643304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:29.627809822 +0000 UTC m=+3246.743729447" watchObservedRunningTime="2026-10-29 23:37:29.635643304 +0000 UTC m=+3246.751562912" Oct 29 23:37:29.995610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:29.989772 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:30.993090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:30.993059 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:31.009719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.009686 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1" path="/var/lib/kubelet/pods/95c9148e-8beb-4ecc-9e98-7e6fc99ebcd1/volumes" Oct 29 23:37:31.603649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.603615 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:31.604632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.604608 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:37:31.696964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.696928 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl"] Oct 29 23:37:31.704688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.704658 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.725049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.725003 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl"] Oct 29 23:37:31.730008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.729979 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51462d\"/\"default-dockercfg-hlpcq\"" Oct 29 23:37:31.730776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.730754 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51462d\"/\"kube-root-ca.crt\"" Oct 29 23:37:31.730943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.730834 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-51462d\"/\"openshift-service-ca.crt\"" Oct 29 23:37:31.731070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.730964 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-51462d\"/\"test-model-51462d-kserve-self-signed-certs\"" Oct 29 23:37:31.810076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.810039 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tls-certs\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.810436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.810384 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-home\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.810852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.810824 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-dshm\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.810989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.810936 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-model-cache\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.811125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.811103 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngs5\" (UniqueName: \"kubernetes.io/projected/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-kube-api-access-xngs5\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.811177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.811145 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tmp-dir\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.912968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.912877 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-home\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.913152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.913049 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-dshm\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.913152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.913084 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-model-cache\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.913152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.913141 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngs5\" (UniqueName: \"kubernetes.io/projected/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-kube-api-access-xngs5\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.913357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.913170 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tmp-dir\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.913357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.913341 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tls-certs\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.913478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.913346 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-home\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.913593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.913551 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-model-cache\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.913802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.913779 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tmp-dir\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.915765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.915739 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-dshm\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.916025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.916006 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tls-certs\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:31.948602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:31.946111 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngs5\" (UniqueName: \"kubernetes.io/projected/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-kube-api-access-xngs5\") pod \"test-model-51462d-kserve-6d946b877f-pr9wl\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:32.041573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.041524 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:32.536981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.536949 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl"] Oct 29 23:37:32.540177 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:37:32.540149 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fef1a8e_2143_4c30_955c_1a6ee0cd3bfb.slice/crio-8e78f09524e68558dcc6f62a8019fc583f35e455e9f8c4953544621431281e41 WatchSource:0}: Error finding container 8e78f09524e68558dcc6f62a8019fc583f35e455e9f8c4953544621431281e41: Status 404 returned error can't find the container with id 8e78f09524e68558dcc6f62a8019fc583f35e455e9f8c4953544621431281e41 Oct 29 23:37:32.615246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.615211 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" event={"ID":"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb","Type":"ContainerStarted","Data":"8e78f09524e68558dcc6f62a8019fc583f35e455e9f8c4953544621431281e41"} Oct 29 23:37:32.845157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.845082 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk"] Oct 29 23:37:32.850034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.850015 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:32.866483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.866457 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:37:32.866691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.866666 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:37:32.872679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.872653 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk"] Oct 29 23:37:32.950518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.950483 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c48be9aa-tvpqk\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:32.950815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.950790 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c48be9aa-tvpqk\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:32.951106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:32.951055 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpnd\" (UniqueName: \"kubernetes.io/projected/593ca3e3-1b25-47b5-9117-9e23838310f8-kube-api-access-wcpnd\") pod \"maas-api-revoke-sub-c48be9aa-tvpqk\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:33.052999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.052961 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c48be9aa-tvpqk\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:33.053394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.053135 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c48be9aa-tvpqk\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:33.053394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.053279 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpnd\" (UniqueName: \"kubernetes.io/projected/593ca3e3-1b25-47b5-9117-9e23838310f8-kube-api-access-wcpnd\") pod \"maas-api-revoke-sub-c48be9aa-tvpqk\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:33.053871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.053849 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c48be9aa-tvpqk\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:33.055363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.055339 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c48be9aa-tvpqk\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:33.082031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.081992 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpnd\" (UniqueName: \"kubernetes.io/projected/593ca3e3-1b25-47b5-9117-9e23838310f8-kube-api-access-wcpnd\") pod \"maas-api-revoke-sub-c48be9aa-tvpqk\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:33.166861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.166782 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:33.486318 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:37:33.486288 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593ca3e3_1b25_47b5_9117_9e23838310f8.slice/crio-c41ba3a8660701370076ae74ca58aaeac63084886e4b2a305ac1f96bf72d4fe9 WatchSource:0}: Error finding container c41ba3a8660701370076ae74ca58aaeac63084886e4b2a305ac1f96bf72d4fe9: Status 404 returned error can't find the container with id c41ba3a8660701370076ae74ca58aaeac63084886e4b2a305ac1f96bf72d4fe9 Oct 29 23:37:33.487141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.487112 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk"] Oct 29 23:37:33.626013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.625977 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" event={"ID":"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb","Type":"ContainerStarted","Data":"a61f481ecc61fc7fe137389413ab5b4c1ba8e600ab269f8f5df71e035bc7caae"} Oct 29 23:37:33.627312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.627287 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" event={"ID":"593ca3e3-1b25-47b5-9117-9e23838310f8","Type":"ContainerStarted","Data":"c41ba3a8660701370076ae74ca58aaeac63084886e4b2a305ac1f96bf72d4fe9"} Oct 29 23:37:33.698647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:33.694549 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" podStartSLOduration=2.694529304 podStartE2EDuration="2.694529304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:33.691989647 +0000 UTC m=+3250.807909253" watchObservedRunningTime="2026-10-29 23:37:33.694529304 +0000 UTC m=+3250.810448911" Oct 29 23:37:34.042213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:34.042181 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:34.654354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:34.654291 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" event={"ID":"593ca3e3-1b25-47b5-9117-9e23838310f8","Type":"ContainerStarted","Data":"d1617e1371a408a7059b6fbfab6645bd4d811a0c4e972318292c42deb048f11e"} Oct 29 23:37:34.696224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:34.696160 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" podStartSLOduration=2.696143105 podStartE2EDuration="2.696143105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:34.693441918 +0000 UTC m=+3251.809361525" watchObservedRunningTime="2026-10-29 23:37:34.696143105 +0000 UTC m=+3251.812062714" Oct 29 23:37:35.412867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:35.412833 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:35.665416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:35.665325 5736 generic.go:356] "Generic (PLEG): container finished" podID="593ca3e3-1b25-47b5-9117-9e23838310f8" containerID="d1617e1371a408a7059b6fbfab6645bd4d811a0c4e972318292c42deb048f11e" exitCode=0 Oct 29 23:37:35.665919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:35.665408 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" event={"ID":"593ca3e3-1b25-47b5-9117-9e23838310f8","Type":"ContainerDied","Data":"d1617e1371a408a7059b6fbfab6645bd4d811a0c4e972318292c42deb048f11e"} Oct 29 23:37:36.859275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:36.859248 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:36.959038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:36.958969 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-xks-ca\") pod \"593ca3e3-1b25-47b5-9117-9e23838310f8\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " Oct 29 23:37:36.959303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:36.959282 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-openshift-ca\") pod \"593ca3e3-1b25-47b5-9117-9e23838310f8\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " Oct 29 23:37:36.959406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:36.959375 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593ca3e3-1b25-47b5-9117-9e23838310f8-kube-api-access-wcpnd\" (UniqueName: \"kubernetes.io/projected/593ca3e3-1b25-47b5-9117-9e23838310f8-kube-api-access-wcpnd\") pod \"593ca3e3-1b25-47b5-9117-9e23838310f8\" (UID: \"593ca3e3-1b25-47b5-9117-9e23838310f8\") " Oct 29 23:37:36.959673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:36.959650 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-openshift-ca" pod "593ca3e3-1b25-47b5-9117-9e23838310f8" (UID: "593ca3e3-1b25-47b5-9117-9e23838310f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:36.960163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:36.960140 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:36.961206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:36.961169 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-xks-ca" pod "593ca3e3-1b25-47b5-9117-9e23838310f8" (UID: "593ca3e3-1b25-47b5-9117-9e23838310f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:37:36.961567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:36.961546 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593ca3e3-1b25-47b5-9117-9e23838310f8-kube-api-access-wcpnd" pod "593ca3e3-1b25-47b5-9117-9e23838310f8" (UID: "593ca3e3-1b25-47b5-9117-9e23838310f8"). InnerVolumeSpecName "kube-api-access-wcpnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:37.061556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:37.061525 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcpnd\" (UniqueName: \"kubernetes.io/projected/593ca3e3-1b25-47b5-9117-9e23838310f8-kube-api-access-wcpnd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:37.061556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:37.061554 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ca3e3-1b25-47b5-9117-9e23838310f8-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:37.686644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:37.686574 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" Oct 29 23:37:37.686828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:37.686636 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk" event={"ID":"593ca3e3-1b25-47b5-9117-9e23838310f8","Type":"ContainerDied","Data":"c41ba3a8660701370076ae74ca58aaeac63084886e4b2a305ac1f96bf72d4fe9"} Oct 29 23:37:37.686828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:37.686691 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41ba3a8660701370076ae74ca58aaeac63084886e4b2a305ac1f96bf72d4fe9" Oct 29 23:37:38.810614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:38.808032 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j"] Oct 29 23:37:38.810614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:38.809671 5736 cpu_manager.go:414] "containerMap: removing container" podUID="593ca3e3-1b25-47b5-9117-9e23838310f8" containerName="revoke-keys" Oct 29 23:37:38.810614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:38.809697 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593ca3e3-1b25-47b5-9117-9e23838310f8" containerName="revoke-keys" Oct 29 23:37:38.810614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:38.809941 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="593ca3e3-1b25-47b5-9117-9e23838310f8" containerName="revoke-keys" Oct 29 23:37:39.085880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.085788 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.117411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.117343 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3b8149-openshift-default-dockercfg-xksvg\"" Oct 29 23:37:39.205763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.191081 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-credential-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.258148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.233776 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j"] Oct 29 23:37:39.258454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.258230 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kdhv\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-kube-api-access-6kdhv\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.258643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.258530 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-certs\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.258747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.258651 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-token\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.258747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.258716 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc2652b-8879-4e8c-8337-ec8faa807dd7-istiod-ca-cert\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.258916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.258792 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-envoy\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.258916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.258854 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-data\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.258916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.258895 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-podinfo\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.259153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.259101 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.360882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.360769 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.361054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.360931 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-credential-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.361132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.361067 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kdhv\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-kube-api-access-6kdhv\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.361132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.361125 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-certs\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.361254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.361202 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-token\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.361315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.361252 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc2652b-8879-4e8c-8337-ec8faa807dd7-istiod-ca-cert\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.361378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.361328 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-envoy\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.361441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.361377 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-data\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.361441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.361413 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-podinfo\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.362334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.362289 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-data\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.362593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.362553 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.363011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.362978 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-certs\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.363122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.363030 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-credential-socket\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.363215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.363175 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc2652b-8879-4e8c-8337-ec8faa807dd7-istiod-ca-cert\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.366879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.366846 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-envoy\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.367359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.367335 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-podinfo\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.389881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.389791 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kdhv\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-kube-api-access-6kdhv\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.422469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.422440 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-token\") pod \"e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:39.572192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:39.572156 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:40.001248 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:37:40.001216 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc2652b_8879_4e8c_8337_ec8faa807dd7.slice/crio-af561e55a0af2480a58916bbdc6e9b332dcfbe384aa3cb4c1c802087aa5b4963 WatchSource:0}: Error finding container af561e55a0af2480a58916bbdc6e9b332dcfbe384aa3cb4c1c802087aa5b4963: Status 404 returned error can't find the container with id af561e55a0af2480a58916bbdc6e9b332dcfbe384aa3cb4c1c802087aa5b4963 Oct 29 23:37:40.001706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:40.001635 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j"] Oct 29 23:37:40.004661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:40.004622 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:37:40.004761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:40.004710 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:37:40.004821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:40.004760 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:37:40.816180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:40.816138 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" event={"ID":"afc2652b-8879-4e8c-8337-ec8faa807dd7","Type":"ContainerStarted","Data":"ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb"} Oct 29 23:37:40.816374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:40.816203 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" event={"ID":"afc2652b-8879-4e8c-8337-ec8faa807dd7","Type":"ContainerStarted","Data":"af561e55a0af2480a58916bbdc6e9b332dcfbe384aa3cb4c1c802087aa5b4963"} Oct 29 23:37:40.865601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:40.864760 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" podStartSLOduration=2.864741757 podStartE2EDuration="2.864741757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:40.864426932 +0000 UTC m=+3257.980346540" watchObservedRunningTime="2026-10-29 23:37:40.864741757 +0000 UTC m=+3257.980661364" Oct 29 23:37:41.582638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:41.581942 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:41.636605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:41.630849 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:41.826048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:41.826012 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:41.826980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:41.826957 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:37:42.042685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:42.042650 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:42.055398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:42.055372 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:42.846256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:42.846227 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:37:46.134678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:46.134636 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4"] Oct 29 23:37:51.135507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.135401 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" podUID="765f41c8-ddd4-4813-b475-eb1723bb2091" containerName="payload-processing" containerID="cri-o://afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a" gracePeriod=25 Oct 29 23:37:51.447714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.447683 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:51.563145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.563107 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-plugins-config-volume\") pod \"765f41c8-ddd4-4813-b475-eb1723bb2091\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " Oct 29 23:37:51.563309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.563173 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-routing-overlay-volume\") pod \"765f41c8-ddd4-4813-b475-eb1723bb2091\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " Oct 29 23:37:51.563309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.563221 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-kube-api-access-xg5tm\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-kube-api-access-xg5tm\") pod \"765f41c8-ddd4-4813-b475-eb1723bb2091\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " Oct 29 23:37:51.563309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.563243 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-provider-credentials\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-provider-credentials\") pod \"765f41c8-ddd4-4813-b475-eb1723bb2091\" (UID: \"765f41c8-ddd4-4813-b475-eb1723bb2091\") " Oct 29 23:37:51.563672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.563637 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-plugins-config-volume" pod "765f41c8-ddd4-4813-b475-eb1723bb2091" (UID: "765f41c8-ddd4-4813-b475-eb1723bb2091"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:51.563780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.563644 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-routing-overlay-volume" pod "765f41c8-ddd4-4813-b475-eb1723bb2091" (UID: "765f41c8-ddd4-4813-b475-eb1723bb2091"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:51.564475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.564437 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:51.564475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.564463 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765f41c8-ddd4-4813-b475-eb1723bb2091-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:51.566250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.566221 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-provider-credentials" pod "765f41c8-ddd4-4813-b475-eb1723bb2091" (UID: "765f41c8-ddd4-4813-b475-eb1723bb2091"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:51.566764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.566734 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-kube-api-access-xg5tm" pod "765f41c8-ddd4-4813-b475-eb1723bb2091" (UID: "765f41c8-ddd4-4813-b475-eb1723bb2091"). InnerVolumeSpecName "kube-api-access-xg5tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:51.666436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.666363 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5tm\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-kube-api-access-xg5tm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:51.666436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:51.666395 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/765f41c8-ddd4-4813-b475-eb1723bb2091-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:37:52.120837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:52.120798 5736 generic.go:356] "Generic (PLEG): container finished" podID="765f41c8-ddd4-4813-b475-eb1723bb2091" containerID="afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a" exitCode=0 Oct 29 23:37:52.120990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:52.120848 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" Oct 29 23:37:52.120990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:52.120878 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" event={"ID":"765f41c8-ddd4-4813-b475-eb1723bb2091","Type":"ContainerDied","Data":"afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a"} Oct 29 23:37:52.120990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:52.120938 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4" event={"ID":"765f41c8-ddd4-4813-b475-eb1723bb2091","Type":"ContainerDied","Data":"5db017de07ea745c770f264c02f643f517dc4f4e0ab647f306ef63fd1ba37a76"} Oct 29 23:37:52.120990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:52.120968 5736 scope.go:122] "RemoveContainer" containerID="afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a" Oct 29 23:37:52.133978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:52.133961 5736 scope.go:122] "RemoveContainer" containerID="afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a" Oct 29 23:37:52.134261 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:52.134239 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a\": container with ID starting with afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a not found: ID does not exist" containerID="afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a" Oct 29 23:37:52.134323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:52.134271 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a"} err="failed to get container status \"afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a\": rpc error: code = NotFound desc = could not find container \"afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a\": container with ID starting with afdf564427c210718a2ae2b7e25493b15ae68d2d9ff44ef1f72710efd9086c4a not found: ID does not exist" Oct 29 23:37:52.192296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:52.192271 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4"] Oct 29 23:37:52.204019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:52.203992 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e1769a/payload-processing-external-model-e2e-extmod-a-e1769a-69666t8s4"] Oct 29 23:37:53.006572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.006536 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765f41c8-ddd4-4813-b475-eb1723bb2091" path="/var/lib/kubelet/pods/765f41c8-ddd4-4813-b475-eb1723bb2091/volumes" Oct 29 23:37:53.367604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.367504 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5"] Oct 29 23:37:53.368628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.368600 5736 cpu_manager.go:414] "containerMap: removing container" podUID="765f41c8-ddd4-4813-b475-eb1723bb2091" containerName="payload-processing" Oct 29 23:37:53.368628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.368625 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765f41c8-ddd4-4813-b475-eb1723bb2091" containerName="payload-processing" Oct 29 23:37:53.368853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.368802 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="765f41c8-ddd4-4813-b475-eb1723bb2091" containerName="payload-processing" Oct 29 23:37:53.374799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.374768 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.411598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.410770 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5"] Oct 29 23:37:53.412458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.411750 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-750972-openshift-default-dockercfg-7wjmt\"" Oct 29 23:37:53.539683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.537224 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6dn\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-kube-api-access-6q6dn\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.539683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.537310 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-certs\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.539683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.537365 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-podinfo\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.539683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.537483 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-envoy\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.539683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.537673 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-data\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.539683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.537825 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istiod-ca-cert\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.539683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.537871 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-token\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.539683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.537946 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.539683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.538159 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-credential-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.700651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.689244 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istiod-ca-cert\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.701162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.701130 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-token\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.701774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.701747 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.702086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.702067 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-credential-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.703467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.703444 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6dn\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-kube-api-access-6q6dn\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.703653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.703636 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-certs\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.703816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.703796 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-podinfo\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.703986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.703971 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-envoy\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.713615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.704261 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-data\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.714617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.713023 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-certs\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.714783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.695465 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istiod-ca-cert\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.714918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.703060 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.715017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.714041 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-data\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.715215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.703304 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-credential-socket\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.760106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.760071 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-token\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.769709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.761429 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6dn\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-kube-api-access-6q6dn\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.783240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.774738 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-podinfo\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.783240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.774954 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-envoy\") pod \"e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:53.997568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:53.997504 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:54.362507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:54.362466 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:37:54.362705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:54.362554 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:37:54.362705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:54.362617 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:37:54.363006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:54.362979 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5"] Oct 29 23:37:55.157342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:55.157304 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" event={"ID":"d69cbd5a-5121-4051-ab5c-7ef57eda66c4","Type":"ContainerStarted","Data":"d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230"} Oct 29 23:37:55.157837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:55.157361 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" event={"ID":"d69cbd5a-5121-4051-ab5c-7ef57eda66c4","Type":"ContainerStarted","Data":"68d066e0b5d9963923c655f9fdec58fa5c8ccee625992adad92d3d928fe47cf7"} Oct 29 23:37:55.241633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:55.236207 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" podStartSLOduration=2.2361891050000002 podStartE2EDuration="2.236189105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:55.234642644 +0000 UTC m=+3272.350562250" watchObservedRunningTime="2026-10-29 23:37:55.236189105 +0000 UTC m=+3272.352108712" Oct 29 23:37:55.998357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:55.998321 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:56.005464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:56.005434 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:56.165905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:56.165877 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:56.166920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:37:56.166896 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:37:56.826247 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:37:56.826210 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:38:01.714257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.714220 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j"] Oct 29 23:38:01.719495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.719453 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.742080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.742048 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j"] Oct 29 23:38:01.742298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.742273 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xggbw\" (UniqueName: \"kubernetes.io/projected/edc14cbd-79a0-449f-a896-eda15ab761c6-kube-api-access-xggbw\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.742378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.742333 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-dshm\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.742435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.742419 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-tmp-dir\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.742479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.742447 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-home\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.742680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.742660 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc14cbd-79a0-449f-a896-eda15ab761c6-tls-certs\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.742753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.742738 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-model-cache\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.748711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.748688 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b7a567\"/\"openshift-service-ca.crt\"" Oct 29 23:38:01.748838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.748725 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b7a567\"/\"kube-root-ca.crt\"" Oct 29 23:38:01.748838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.748749 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b7a567\"/\"default-dockercfg-q2rtg\"" Oct 29 23:38:01.748838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.748780 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b7a567\"/\"test-model-b7a567-kserve-self-signed-certs\"" Oct 29 23:38:01.844166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.844132 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xggbw\" (UniqueName: \"kubernetes.io/projected/edc14cbd-79a0-449f-a896-eda15ab761c6-kube-api-access-xggbw\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.844354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.844223 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-dshm\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.844354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.844259 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-tmp-dir\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.844354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.844291 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-home\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.844553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.844531 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc14cbd-79a0-449f-a896-eda15ab761c6-tls-certs\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.844725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.844700 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-model-cache\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.844803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.844762 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-tmp-dir\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.844850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.844803 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-home\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.845255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.845098 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-model-cache\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.847174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.847148 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-dshm\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.847422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.847400 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc14cbd-79a0-449f-a896-eda15ab761c6-tls-certs\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:01.869705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:01.869670 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xggbw\" (UniqueName: \"kubernetes.io/projected/edc14cbd-79a0-449f-a896-eda15ab761c6-kube-api-access-xggbw\") pod \"test-model-b7a567-kserve-7b8966f899-w5z4j\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:02.039439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:02.038016 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:02.619605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:02.617665 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j"] Oct 29 23:38:03.255225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:03.255185 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" event={"ID":"edc14cbd-79a0-449f-a896-eda15ab761c6","Type":"ContainerStarted","Data":"ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171"} Oct 29 23:38:03.255684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:03.255235 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" event={"ID":"edc14cbd-79a0-449f-a896-eda15ab761c6","Type":"ContainerStarted","Data":"438bf51db92c2a4df0394d101be4251d5b828a258e6d522f3a22412bd3303ab1"} Oct 29 23:38:03.299347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:03.299293 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" podStartSLOduration=2.299279161 podStartE2EDuration="2.299279161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:03.295680949 +0000 UTC m=+3280.411600574" watchObservedRunningTime="2026-10-29 23:38:03.299279161 +0000 UTC m=+3280.415198776" Oct 29 23:38:04.038924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:04.038890 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:12.038806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:12.038770 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:12.056456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:12.056427 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:12.358919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:12.358819 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:38:24.019791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:24.019759 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:38:24.052040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:24.052009 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:38:26.466160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:26.466116 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r"] Oct 29 23:38:26.481115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:26.481072 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" containerID="cri-o://26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914" gracePeriod=30 Oct 29 23:38:26.826375 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:38:26.826287 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:38:28.078362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.078316 5736 scope.go:122] "RemoveContainer" containerID="5b2e2db7688392c66614a1a4f0e4d00aa78d4e91468928f59138ee92b0303fe4" Oct 29 23:38:28.258754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.258721 5736 scope.go:122] "RemoveContainer" containerID="fe581187574050876cb1382b73f452c1a0dfb07053e00796fada1dc44b33e476" Oct 29 23:38:28.313880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.313836 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 29 23:38:28.332718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.327000 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.333853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.333822 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.360334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.350660 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" podUID="55389672-b516-4422-b514-3ca1355c361a" containerName="main" probeResult="failure" output="Get \"https://10.134.1.103:8000/health\": context deadline exceeded" Oct 29 23:38:28.360334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.350792 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" podUID="edc14cbd-79a0-449f-a896-eda15ab761c6" containerName="main" probeResult="failure" output="Get \"https://10.134.1.112:8000/ready\": context deadline exceeded" Oct 29 23:38:28.371379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.371348 5736 scope.go:122] "RemoveContainer" containerID="47f12046a58bcc082ab599a172cf85f257c7d999cb240d5d0680785368e50f39" Oct 29 23:38:28.398842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.398800 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.403101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.399074 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 23:38:28.413174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.399146 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.414395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.402652 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.424805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.402682 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 23:38:28.425246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.402701 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 29 23:38:28.425623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.402730 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.425955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.412793 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 29 23:38:28.447417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.445760 5736 scope.go:122] "RemoveContainer" containerID="b27b14d1b548cbea4aebed7c3c13b84723533d44fa01a5132d771ab44d5aa652" Oct 29 23:38:28.447417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.446279 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" podUID="55389672-b516-4422-b514-3ca1355c361a" containerName="main" probeResult="failure" output="Get \"https://10.134.1.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.541680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.525042 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.712598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.712390 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 29 23:38:28.717141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.717109 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 29 23:38:28.766071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.765896 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g" podUID="ff018793-fe72-4680-af0f-6a7b1d4b94b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.861821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.854878 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" podUID="2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" containerName="main" probeResult="failure" output="Get \"https://10.134.1.108:8000/ready\": context deadline exceeded" Oct 29 23:38:28.867984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.867944 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.879883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.879842 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:28.959694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:28.953689 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:38:31.906133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.906100 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6"] Oct 29 23:38:31.911380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.911352 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:31.915052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.915025 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:38:31.938141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.938040 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:38:31.938141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.938050 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:38:31.943148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.943123 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6"] Oct 29 23:38:31.995668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.995425 5736 generic.go:356] "Generic (PLEG): container finished" podID="ccdaaaba-4830-4237-9850-094ec264d63d" containerID="26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914" exitCode=0 Oct 29 23:38:31.995668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.995475 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" Oct 29 23:38:31.995668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.995514 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" event={"ID":"ccdaaaba-4830-4237-9850-094ec264d63d","Type":"ContainerDied","Data":"26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914"} Oct 29 23:38:31.995668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.995575 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r" event={"ID":"ccdaaaba-4830-4237-9850-094ec264d63d","Type":"ContainerDied","Data":"b97906f51447f98fa9a912d1ac551e476d3f34e11bc40170fbb1d8c9bb84e186"} Oct 29 23:38:31.996049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:31.995787 5736 scope.go:122] "RemoveContainer" containerID="26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914" Oct 29 23:38:32.002352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.002324 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccdaaaba-4830-4237-9850-094ec264d63d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdaaaba-4830-4237-9850-094ec264d63d-istiod-ca-cert\") pod \"ccdaaaba-4830-4237-9850-094ec264d63d\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " Oct 29 23:38:32.002475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.002393 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-istio-token\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-istio-token\") pod \"ccdaaaba-4830-4237-9850-094ec264d63d\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " Oct 29 23:38:32.002475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.002423 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-envoy\") pod \"ccdaaaba-4830-4237-9850-094ec264d63d\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " Oct 29 23:38:32.002639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.002511 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-certs\") pod \"ccdaaaba-4830-4237-9850-094ec264d63d\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " Oct 29 23:38:32.002639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.002544 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-credential-socket\") pod \"ccdaaaba-4830-4237-9850-094ec264d63d\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " Oct 29 23:38:32.002639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.002607 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-data\") pod \"ccdaaaba-4830-4237-9850-094ec264d63d\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " Oct 29 23:38:32.002801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.002753 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccdaaaba-4830-4237-9850-094ec264d63d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdaaaba-4830-4237-9850-094ec264d63d-istio-podinfo\") pod \"ccdaaaba-4830-4237-9850-094ec264d63d\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " Oct 29 23:38:32.002864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.002825 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-socket\") pod \"ccdaaaba-4830-4237-9850-094ec264d63d\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " Oct 29 23:38:32.002922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.002859 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-kube-api-access-hf8tw\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-kube-api-access-hf8tw\") pod \"ccdaaaba-4830-4237-9850-094ec264d63d\" (UID: \"ccdaaaba-4830-4237-9850-094ec264d63d\") " Oct 29 23:38:32.003547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.003062 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-credential-socket" pod "ccdaaaba-4830-4237-9850-094ec264d63d" (UID: "ccdaaaba-4830-4237-9850-094ec264d63d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:32.003547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.003360 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr6fc\" (UniqueName: \"kubernetes.io/projected/6fa40060-1336-4b82-96aa-e1cb620c4ec5-kube-api-access-nr6fc\") pod \"maas-api-revoke-sub-6dc10bf0-rkhc6\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.003547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.003432 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc10bf0-rkhc6\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.003797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.003654 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-certs" pod "ccdaaaba-4830-4237-9850-094ec264d63d" (UID: "ccdaaaba-4830-4237-9850-094ec264d63d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:32.003855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.003790 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc10bf0-rkhc6\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.003962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.003936 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-data" pod "ccdaaaba-4830-4237-9850-094ec264d63d" (UID: "ccdaaaba-4830-4237-9850-094ec264d63d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:32.004021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.003967 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-socket" pod "ccdaaaba-4830-4237-9850-094ec264d63d" (UID: "ccdaaaba-4830-4237-9850-094ec264d63d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:32.004136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.004117 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:32.004202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.004142 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:32.004202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.004156 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:32.004202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.004169 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:32.004644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.004613 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccdaaaba-4830-4237-9850-094ec264d63d-istiod-ca-cert" pod "ccdaaaba-4830-4237-9850-094ec264d63d" (UID: "ccdaaaba-4830-4237-9850-094ec264d63d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:32.009612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.006521 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-kube-api-access-hf8tw" pod "ccdaaaba-4830-4237-9850-094ec264d63d" (UID: "ccdaaaba-4830-4237-9850-094ec264d63d"). InnerVolumeSpecName "kube-api-access-hf8tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:32.009612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.008991 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccdaaaba-4830-4237-9850-094ec264d63d-istio-podinfo" pod "ccdaaaba-4830-4237-9850-094ec264d63d" (UID: "ccdaaaba-4830-4237-9850-094ec264d63d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:38:32.009612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.009004 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-istio-token" pod "ccdaaaba-4830-4237-9850-094ec264d63d" (UID: "ccdaaaba-4830-4237-9850-094ec264d63d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:32.009612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.009103 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-envoy" pod "ccdaaaba-4830-4237-9850-094ec264d63d" (UID: "ccdaaaba-4830-4237-9850-094ec264d63d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:32.014328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.014297 5736 scope.go:122] "RemoveContainer" containerID="26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914" Oct 29 23:38:32.014697 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:38:32.014672 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914\": container with ID starting with 26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914 not found: ID does not exist" containerID="26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914" Oct 29 23:38:32.014810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.014703 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914"} err="failed to get container status \"26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914\": rpc error: code = NotFound desc = could not find container \"26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914\": container with ID starting with 26fb0e49816bbb0357bbc735663c57e77dcd0df310449b3e75e063befbb6c914 not found: ID does not exist" Oct 29 23:38:32.115509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.113644 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc10bf0-rkhc6\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.117855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.116521 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc10bf0-rkhc6\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.117855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.117012 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc10bf0-rkhc6\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.117855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.117471 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr6fc\" (UniqueName: \"kubernetes.io/projected/6fa40060-1336-4b82-96aa-e1cb620c4ec5-kube-api-access-nr6fc\") pod \"maas-api-revoke-sub-6dc10bf0-rkhc6\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.117855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.117658 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdaaaba-4830-4237-9850-094ec264d63d-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:32.117855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.117681 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdaaaba-4830-4237-9850-094ec264d63d-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:32.117855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.117694 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf8tw\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-kube-api-access-hf8tw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:32.117855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.117710 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdaaaba-4830-4237-9850-094ec264d63d-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:32.117855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.117722 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdaaaba-4830-4237-9850-094ec264d63d-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:32.120712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.120686 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc10bf0-rkhc6\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.142158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.142127 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr6fc\" (UniqueName: \"kubernetes.io/projected/6fa40060-1336-4b82-96aa-e1cb620c4ec5-kube-api-access-nr6fc\") pod \"maas-api-revoke-sub-6dc10bf0-rkhc6\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.225714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.225676 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:32.357624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.355995 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r"] Oct 29 23:38:32.362754 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.362700 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-rz49r"] Oct 29 23:38:32.538911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:32.538874 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6"] Oct 29 23:38:32.540368 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:38:32.540338 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa40060_1336_4b82_96aa_e1cb620c4ec5.slice/crio-812e7d52ad246ae5f30ebff22ff71adab03296ca08178335d321bff9a38b6d5d WatchSource:0}: Error finding container 812e7d52ad246ae5f30ebff22ff71adab03296ca08178335d321bff9a38b6d5d: Status 404 returned error can't find the container with id 812e7d52ad246ae5f30ebff22ff71adab03296ca08178335d321bff9a38b6d5d Oct 29 23:38:33.007862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:33.007828 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" path="/var/lib/kubelet/pods/ccdaaaba-4830-4237-9850-094ec264d63d/volumes" Oct 29 23:38:33.008344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:33.008303 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" event={"ID":"6fa40060-1336-4b82-96aa-e1cb620c4ec5","Type":"ContainerStarted","Data":"812e7d52ad246ae5f30ebff22ff71adab03296ca08178335d321bff9a38b6d5d"} Oct 29 23:38:34.019751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:34.019674 5736 generic.go:356] "Generic (PLEG): container finished" podID="6fa40060-1336-4b82-96aa-e1cb620c4ec5" containerID="807b0396f65a3d2036bf77f044ea969c37a8ca879105663e1558e49e787217c7" exitCode=0 Oct 29 23:38:34.019751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:34.019740 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" event={"ID":"6fa40060-1336-4b82-96aa-e1cb620c4ec5","Type":"ContainerDied","Data":"807b0396f65a3d2036bf77f044ea969c37a8ca879105663e1558e49e787217c7"} Oct 29 23:38:34.030271 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:34.030223 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj"] Oct 29 23:38:34.039034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:34.039005 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b88163e-dzgjj"] Oct 29 23:38:35.070169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.070129 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2bc491-821c-45ee-9068-87d35cd70334" path="/var/lib/kubelet/pods/1d2bc491-821c-45ee-9068-87d35cd70334/volumes" Oct 29 23:38:35.246163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.246133 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:35.379474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.379394 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-openshift-ca\") pod \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " Oct 29 23:38:35.379667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.379505 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-xks-ca\") pod \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " Oct 29 23:38:35.379734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.379673 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa40060-1336-4b82-96aa-e1cb620c4ec5-kube-api-access-nr6fc\" (UniqueName: \"kubernetes.io/projected/6fa40060-1336-4b82-96aa-e1cb620c4ec5-kube-api-access-nr6fc\") pod \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\" (UID: \"6fa40060-1336-4b82-96aa-e1cb620c4ec5\") " Oct 29 23:38:35.379796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.379744 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-openshift-ca" pod "6fa40060-1336-4b82-96aa-e1cb620c4ec5" (UID: "6fa40060-1336-4b82-96aa-e1cb620c4ec5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:35.380341 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.380322 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:35.381638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.381608 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-xks-ca" pod "6fa40060-1336-4b82-96aa-e1cb620c4ec5" (UID: "6fa40060-1336-4b82-96aa-e1cb620c4ec5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:38:35.381853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.381828 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa40060-1336-4b82-96aa-e1cb620c4ec5-kube-api-access-nr6fc" pod "6fa40060-1336-4b82-96aa-e1cb620c4ec5" (UID: "6fa40060-1336-4b82-96aa-e1cb620c4ec5"). InnerVolumeSpecName "kube-api-access-nr6fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:35.481687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.481652 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa40060-1336-4b82-96aa-e1cb620c4ec5-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:35.481687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:35.481690 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr6fc\" (UniqueName: \"kubernetes.io/projected/6fa40060-1336-4b82-96aa-e1cb620c4ec5-kube-api-access-nr6fc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:36.082284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:36.082247 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" Oct 29 23:38:36.082284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:36.082258 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6" event={"ID":"6fa40060-1336-4b82-96aa-e1cb620c4ec5","Type":"ContainerDied","Data":"812e7d52ad246ae5f30ebff22ff71adab03296ca08178335d321bff9a38b6d5d"} Oct 29 23:38:36.082748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:36.082316 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="812e7d52ad246ae5f30ebff22ff71adab03296ca08178335d321bff9a38b6d5d" Oct 29 23:38:42.716421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.716385 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q"] Oct 29 23:38:42.717280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.717261 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6fa40060-1336-4b82-96aa-e1cb620c4ec5" containerName="revoke-keys" Oct 29 23:38:42.717280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.717281 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa40060-1336-4b82-96aa-e1cb620c4ec5" containerName="revoke-keys" Oct 29 23:38:42.717361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.717296 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" Oct 29 23:38:42.717361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.717303 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" Oct 29 23:38:42.717461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.717452 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa40060-1336-4b82-96aa-e1cb620c4ec5" containerName="revoke-keys" Oct 29 23:38:42.717500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.717465 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccdaaaba-4830-4237-9850-094ec264d63d" containerName="istio-proxy" Oct 29 23:38:42.721339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.721321 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:42.748960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.748926 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:38:42.761158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.761133 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:38:42.768454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.768423 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q"] Oct 29 23:38:42.803038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.803010 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ad63b16-9tg2q\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:42.803218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.803192 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ad63b16-9tg2q\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:42.803335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.803319 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzqf\" (UniqueName: \"kubernetes.io/projected/a38a56c6-a367-4443-b2d7-c8777d8a4e21-kube-api-access-2dzqf\") pod \"maas-api-revoke-sub-7ad63b16-9tg2q\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:42.904848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.904822 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ad63b16-9tg2q\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:42.905004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.904933 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ad63b16-9tg2q\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:42.905004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.904991 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzqf\" (UniqueName: \"kubernetes.io/projected/a38a56c6-a367-4443-b2d7-c8777d8a4e21-kube-api-access-2dzqf\") pod \"maas-api-revoke-sub-7ad63b16-9tg2q\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:42.905528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.905502 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ad63b16-9tg2q\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:42.907848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.907826 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ad63b16-9tg2q\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:42.927826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:42.927805 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzqf\" (UniqueName: \"kubernetes.io/projected/a38a56c6-a367-4443-b2d7-c8777d8a4e21-kube-api-access-2dzqf\") pod \"maas-api-revoke-sub-7ad63b16-9tg2q\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:43.053318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:43.053247 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:43.363692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:43.363662 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q"] Oct 29 23:38:43.364950 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:38:43.364920 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38a56c6_a367_4443_b2d7_c8777d8a4e21.slice/crio-5f06ed51a99b48bc62b1ad65346321aac31fb60908ec03c259c76e3feb0cc60e WatchSource:0}: Error finding container 5f06ed51a99b48bc62b1ad65346321aac31fb60908ec03c259c76e3feb0cc60e: Status 404 returned error can't find the container with id 5f06ed51a99b48bc62b1ad65346321aac31fb60908ec03c259c76e3feb0cc60e Oct 29 23:38:44.160081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:44.159996 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" event={"ID":"a38a56c6-a367-4443-b2d7-c8777d8a4e21","Type":"ContainerStarted","Data":"4a55adeae65847c678366790ca80235cf5366402cb2b5dafdee3647ef2ec14c5"} Oct 29 23:38:44.160081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:44.160048 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" event={"ID":"a38a56c6-a367-4443-b2d7-c8777d8a4e21","Type":"ContainerStarted","Data":"5f06ed51a99b48bc62b1ad65346321aac31fb60908ec03c259c76e3feb0cc60e"} Oct 29 23:38:44.202973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:44.202914 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" podStartSLOduration=2.202899187 podStartE2EDuration="2.202899187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:44.199674487 +0000 UTC m=+3321.315594092" watchObservedRunningTime="2026-10-29 23:38:44.202899187 +0000 UTC m=+3321.318818792" Oct 29 23:38:45.170371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:45.170339 5736 generic.go:356] "Generic (PLEG): container finished" podID="a38a56c6-a367-4443-b2d7-c8777d8a4e21" containerID="4a55adeae65847c678366790ca80235cf5366402cb2b5dafdee3647ef2ec14c5" exitCode=0 Oct 29 23:38:45.170823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:45.170426 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" event={"ID":"a38a56c6-a367-4443-b2d7-c8777d8a4e21","Type":"ContainerDied","Data":"4a55adeae65847c678366790ca80235cf5366402cb2b5dafdee3647ef2ec14c5"} Oct 29 23:38:46.364635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.364574 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:46.468753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.468724 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-openshift-ca\") pod \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " Oct 29 23:38:46.468924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.468836 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-xks-ca\") pod \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " Oct 29 23:38:46.468924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.468870 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38a56c6-a367-4443-b2d7-c8777d8a4e21-kube-api-access-2dzqf\" (UniqueName: \"kubernetes.io/projected/a38a56c6-a367-4443-b2d7-c8777d8a4e21-kube-api-access-2dzqf\") pod \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\" (UID: \"a38a56c6-a367-4443-b2d7-c8777d8a4e21\") " Oct 29 23:38:46.469130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.469103 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-openshift-ca" pod "a38a56c6-a367-4443-b2d7-c8777d8a4e21" (UID: "a38a56c6-a367-4443-b2d7-c8777d8a4e21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:46.469553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.469531 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:46.471156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.471129 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-xks-ca" pod "a38a56c6-a367-4443-b2d7-c8777d8a4e21" (UID: "a38a56c6-a367-4443-b2d7-c8777d8a4e21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:38:46.471257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.471139 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38a56c6-a367-4443-b2d7-c8777d8a4e21-kube-api-access-2dzqf" pod "a38a56c6-a367-4443-b2d7-c8777d8a4e21" (UID: "a38a56c6-a367-4443-b2d7-c8777d8a4e21"). InnerVolumeSpecName "kube-api-access-2dzqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:46.571238 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.571211 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38a56c6-a367-4443-b2d7-c8777d8a4e21-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:46.571238 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:46.571241 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dzqf\" (UniqueName: \"kubernetes.io/projected/a38a56c6-a367-4443-b2d7-c8777d8a4e21-kube-api-access-2dzqf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:47.192838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:47.192806 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" Oct 29 23:38:47.193112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:47.192843 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q" event={"ID":"a38a56c6-a367-4443-b2d7-c8777d8a4e21","Type":"ContainerDied","Data":"5f06ed51a99b48bc62b1ad65346321aac31fb60908ec03c259c76e3feb0cc60e"} Oct 29 23:38:47.193246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:47.193151 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f06ed51a99b48bc62b1ad65346321aac31fb60908ec03c259c76e3feb0cc60e" Oct 29 23:38:49.405948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.405912 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8"] Oct 29 23:38:49.407044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.407020 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a38a56c6-a367-4443-b2d7-c8777d8a4e21" containerName="revoke-keys" Oct 29 23:38:49.407044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.407042 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38a56c6-a367-4443-b2d7-c8777d8a4e21" containerName="revoke-keys" Oct 29 23:38:49.407234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.407226 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38a56c6-a367-4443-b2d7-c8777d8a4e21" containerName="revoke-keys" Oct 29 23:38:49.413594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.411508 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.434690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.434663 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8"] Oct 29 23:38:49.441636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.441614 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-51462d-openshift-default-dockercfg-rt7w7\"" Oct 29 23:38:49.456047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.456010 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-certs\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.456283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.456249 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-token\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.456432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.456411 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-podinfo\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.456503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.456460 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.456640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.456567 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-credential-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.456778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.456756 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1712a1a-9e91-4570-995c-20e50ade63bf-istiod-ca-cert\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.456847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.456805 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-envoy\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.456916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.456895 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6j8s\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-kube-api-access-r6j8s\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.456982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.456955 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-data\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.558676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.558639 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-credential-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.558837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.558760 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1712a1a-9e91-4570-995c-20e50ade63bf-istiod-ca-cert\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.558837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.558799 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-envoy\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.558843 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6j8s\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-kube-api-access-r6j8s\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.558887 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-data\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.558960 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-certs\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.559031 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-credential-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.559182 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-token\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.559254 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-data\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.559313 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-podinfo\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559462 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.559343 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-certs\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559462 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.559349 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.559470 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1712a1a-9e91-4570-995c-20e50ade63bf-istiod-ca-cert\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.559659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.559625 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.561101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.561079 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-envoy\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.561633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.561610 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-podinfo\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.581967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.581941 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-token\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.582132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.582111 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6j8s\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-kube-api-access-r6j8s\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:49.728588 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:49.728556 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:50.056513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:50.056488 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8"] Oct 29 23:38:50.057668 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:38:50.057641 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1712a1a_9e91_4570_995c_20e50ade63bf.slice/crio-8d82b8f381b62857f6663a013e7ae60b172c503633bc02e9a4505482162e3b1b WatchSource:0}: Error finding container 8d82b8f381b62857f6663a013e7ae60b172c503633bc02e9a4505482162e3b1b: Status 404 returned error can't find the container with id 8d82b8f381b62857f6663a013e7ae60b172c503633bc02e9a4505482162e3b1b Oct 29 23:38:50.068961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:50.063936 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:38:50.068961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:50.064168 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:38:50.068961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:50.064261 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:38:50.278037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:50.277994 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" event={"ID":"d1712a1a-9e91-4570-995c-20e50ade63bf","Type":"ContainerStarted","Data":"17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f"} Oct 29 23:38:50.278210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:50.278094 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" event={"ID":"d1712a1a-9e91-4570-995c-20e50ade63bf","Type":"ContainerStarted","Data":"8d82b8f381b62857f6663a013e7ae60b172c503633bc02e9a4505482162e3b1b"} Oct 29 23:38:50.324679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:50.324496 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" podStartSLOduration=1.324480388 podStartE2EDuration="1.324480388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:50.321879807 +0000 UTC m=+3327.437799415" watchObservedRunningTime="2026-10-29 23:38:50.324480388 +0000 UTC m=+3327.440399995" Oct 29 23:38:50.729286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:50.729250 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:51.753604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:51.744702 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 29 23:38:51.753604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:51.744770 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 29 23:38:52.728905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:52.728871 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 29 23:38:52.729067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:52.728925 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 29 23:38:53.729464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:53.729426 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 29 23:38:53.729908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:53.729491 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 29 23:38:54.388463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:54.388428 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk"] Oct 29 23:38:54.735605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:54.732000 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 29 23:38:54.735605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:54.732065 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 29 23:38:55.226523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:55.226481 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb"] Oct 29 23:38:55.729008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:55.728969 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 29 23:38:55.729314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:55.729283 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 29 23:38:56.729186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:56.729150 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body= Oct 29 23:38:56.729599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:56.729211 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" Oct 29 23:38:56.826223 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:38:56.826172 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:38:57.735004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:57.734972 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:57.735487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:57.735328 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:57.735893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:57.735875 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:38:59.391520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.391470 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" podUID="b7eb2bc1-ff1e-4378-9e92-f6d120533f81" containerName="payload-processing" containerID="cri-o://2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e" gracePeriod=25 Oct 29 23:38:59.696741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.696683 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:38:59.812783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.812744 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-kube-api-access-g48zb\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-kube-api-access-g48zb\") pod \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " Oct 29 23:38:59.812959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.812801 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-plugins-config-volume\") pod \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " Oct 29 23:38:59.812959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.812871 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-routing-overlay-volume\") pod \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " Oct 29 23:38:59.813065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.812997 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-provider-credentials\") pod \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\" (UID: \"b7eb2bc1-ff1e-4378-9e92-f6d120533f81\") " Oct 29 23:38:59.813273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.813244 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-plugins-config-volume" pod "b7eb2bc1-ff1e-4378-9e92-f6d120533f81" (UID: "b7eb2bc1-ff1e-4378-9e92-f6d120533f81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:59.813382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.813269 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-routing-overlay-volume" pod "b7eb2bc1-ff1e-4378-9e92-f6d120533f81" (UID: "b7eb2bc1-ff1e-4378-9e92-f6d120533f81"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:59.813768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.813745 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:59.813768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.813767 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:59.815182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.815158 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-provider-credentials" pod "b7eb2bc1-ff1e-4378-9e92-f6d120533f81" (UID: "b7eb2bc1-ff1e-4378-9e92-f6d120533f81"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:59.815313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.815194 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-kube-api-access-g48zb" pod "b7eb2bc1-ff1e-4378-9e92-f6d120533f81" (UID: "b7eb2bc1-ff1e-4378-9e92-f6d120533f81"). InnerVolumeSpecName "kube-api-access-g48zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:59.915639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.915598 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48zb\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-kube-api-access-g48zb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:38:59.915639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:38:59.915641 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7eb2bc1-ff1e-4378-9e92-f6d120533f81-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:00.227711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.227673 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" podUID="58d8f67a-d528-4007-b234-dce4411fa198" containerName="payload-processing" containerID="cri-o://528efff55020c67ef6369ca36490c62957beff32e058df1e4e9b1079d0c6c06e" gracePeriod=25 Oct 29 23:39:00.419699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.419641 5736 generic.go:356] "Generic (PLEG): container finished" podID="b7eb2bc1-ff1e-4378-9e92-f6d120533f81" containerID="2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e" exitCode=0 Oct 29 23:39:00.420201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.419772 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" event={"ID":"b7eb2bc1-ff1e-4378-9e92-f6d120533f81","Type":"ContainerDied","Data":"2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e"} Oct 29 23:39:00.420292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.420254 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" event={"ID":"b7eb2bc1-ff1e-4378-9e92-f6d120533f81","Type":"ContainerDied","Data":"b036336974282e3444d49a0138077f0473de3797cd73ddf5d17212012eb21b75"} Oct 29 23:39:00.420292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.419854 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk" Oct 29 23:39:00.420412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.420294 5736 scope.go:122] "RemoveContainer" containerID="2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e" Oct 29 23:39:00.433957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.433781 5736 generic.go:356] "Generic (PLEG): container finished" podID="58d8f67a-d528-4007-b234-dce4411fa198" containerID="528efff55020c67ef6369ca36490c62957beff32e058df1e4e9b1079d0c6c06e" exitCode=0 Oct 29 23:39:00.433957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.433898 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" event={"ID":"58d8f67a-d528-4007-b234-dce4411fa198","Type":"ContainerDied","Data":"528efff55020c67ef6369ca36490c62957beff32e058df1e4e9b1079d0c6c06e"} Oct 29 23:39:00.460612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.460566 5736 scope.go:122] "RemoveContainer" containerID="2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e" Oct 29 23:39:00.461124 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:39:00.461038 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e\": container with ID starting with 2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e not found: ID does not exist" containerID="2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e" Oct 29 23:39:00.461124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.461078 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e"} err="failed to get container status \"2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e\": rpc error: code = NotFound desc = could not find container \"2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e\": container with ID starting with 2e74773251887d54e9d207c9b828c97bd4b586a4eed9b4a825a82d9923cf849e not found: ID does not exist" Oct 29 23:39:00.503541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.503512 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk"] Oct 29 23:39:00.512862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.512838 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a84df8/payload-processing-external-model-e2e-extmod-a-a84df8-568bhkchk"] Oct 29 23:39:00.528949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.528926 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:39:00.631624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.631573 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-routing-overlay-volume\") pod \"58d8f67a-d528-4007-b234-dce4411fa198\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " Oct 29 23:39:00.631795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.631779 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-provider-credentials\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-provider-credentials\") pod \"58d8f67a-d528-4007-b234-dce4411fa198\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " Oct 29 23:39:00.632045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.632018 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-routing-overlay-volume" pod "58d8f67a-d528-4007-b234-dce4411fa198" (UID: "58d8f67a-d528-4007-b234-dce4411fa198"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:39:00.632091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.632033 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-kube-api-access-w86ns\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-kube-api-access-w86ns\") pod \"58d8f67a-d528-4007-b234-dce4411fa198\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " Oct 29 23:39:00.632169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.632150 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-plugins-config-volume\") pod \"58d8f67a-d528-4007-b234-dce4411fa198\" (UID: \"58d8f67a-d528-4007-b234-dce4411fa198\") " Oct 29 23:39:00.632559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.632532 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-plugins-config-volume" pod "58d8f67a-d528-4007-b234-dce4411fa198" (UID: "58d8f67a-d528-4007-b234-dce4411fa198"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:39:00.633242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.633217 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:00.633300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.633249 5736 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58d8f67a-d528-4007-b234-dce4411fa198-routing-overlay-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:00.634132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.634105 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-provider-credentials" pod "58d8f67a-d528-4007-b234-dce4411fa198" (UID: "58d8f67a-d528-4007-b234-dce4411fa198"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:39:00.634243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.634146 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-kube-api-access-w86ns" pod "58d8f67a-d528-4007-b234-dce4411fa198" (UID: "58d8f67a-d528-4007-b234-dce4411fa198"). InnerVolumeSpecName "kube-api-access-w86ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:39:00.734965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.734928 5736 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-provider-credentials\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:00.734965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:00.734962 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w86ns\" (UniqueName: \"kubernetes.io/projected/58d8f67a-d528-4007-b234-dce4411fa198-kube-api-access-w86ns\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:01.010496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:01.010456 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7eb2bc1-ff1e-4378-9e92-f6d120533f81" path="/var/lib/kubelet/pods/b7eb2bc1-ff1e-4378-9e92-f6d120533f81/volumes" Oct 29 23:39:01.446984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:01.446900 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" event={"ID":"58d8f67a-d528-4007-b234-dce4411fa198","Type":"ContainerDied","Data":"cec243e7a5af48ae4636905209bb126227f8bd23450d2227796db1e861a8258c"} Oct 29 23:39:01.446984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:01.446969 5736 scope.go:122] "RemoveContainer" containerID="528efff55020c67ef6369ca36490c62957beff32e058df1e4e9b1079d0c6c06e" Oct 29 23:39:01.447459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:01.446915 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb" Oct 29 23:39:01.517865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:01.517830 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb"] Oct 29 23:39:01.530235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:01.530181 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e2ac7/payload-processing-external-model-e2e-extmod-b-4e2ac7-79dcqvxgb"] Oct 29 23:39:03.037081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:03.036865 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d8f67a-d528-4007-b234-dce4411fa198" path="/var/lib/kubelet/pods/58d8f67a-d528-4007-b234-dce4411fa198/volumes" Oct 29 23:39:13.563228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.563141 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z"] Oct 29 23:39:13.564065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.564044 5736 cpu_manager.go:414] "containerMap: removing container" podUID="58d8f67a-d528-4007-b234-dce4411fa198" containerName="payload-processing" Oct 29 23:39:13.564124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.564067 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d8f67a-d528-4007-b234-dce4411fa198" containerName="payload-processing" Oct 29 23:39:13.564124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.564080 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b7eb2bc1-ff1e-4378-9e92-f6d120533f81" containerName="payload-processing" Oct 29 23:39:13.564124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.564085 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7eb2bc1-ff1e-4378-9e92-f6d120533f81" containerName="payload-processing" Oct 29 23:39:13.564252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.564242 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d8f67a-d528-4007-b234-dce4411fa198" containerName="payload-processing" Oct 29 23:39:13.564290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.564253 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7eb2bc1-ff1e-4378-9e92-f6d120533f81" containerName="payload-processing" Oct 29 23:39:13.569681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.569661 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.591479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.591443 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z"] Oct 29 23:39:13.596907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.596883 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-750972-serving-cert\"" Oct 29 23:39:13.670549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.670512 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64p4l\" (UniqueName: \"kubernetes.io/projected/64e5d601-93ef-4424-b06c-bc5ca97943e5-kube-api-access-64p4l\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.670753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.670612 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.670868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.670838 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-metrics-certs\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.671004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.670985 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5d601-93ef-4424-b06c-bc5ca97943e5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.785261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.785207 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64p4l\" (UniqueName: \"kubernetes.io/projected/64e5d601-93ef-4424-b06c-bc5ca97943e5-kube-api-access-64p4l\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.785641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.785608 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.785997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.785974 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-metrics-certs\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.786248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.786229 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5d601-93ef-4424-b06c-bc5ca97943e5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.794602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.787425 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5d601-93ef-4424-b06c-bc5ca97943e5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.794824 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:39:13.788435 5736 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-serving-cert: secret "maas-api-e2e-inf-a-750972-serving-cert" not found Oct 29 23:39:13.795008 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:39:13.794992 5736 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls podName:64e5d601-93ef-4424-b06c-bc5ca97943e5 nodeName:}" failed. No retries permitted until 2026-10-29 23:39:14.294965482 +0000 UTC m=+3351.410885083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls") pod "maas-api-e2e-inf-a-750972-9bf947464-z9l7z" (UID: "64e5d601-93ef-4424-b06c-bc5ca97943e5") : secret "maas-api-e2e-inf-a-750972-serving-cert" not found Oct 29 23:39:13.829001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.828291 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64p4l\" (UniqueName: \"kubernetes.io/projected/64e5d601-93ef-4424-b06c-bc5ca97943e5-kube-api-access-64p4l\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:13.833645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:13.831378 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-metrics-certs\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:14.295138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:14.295096 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:14.297762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:14.297727 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls\") pod \"maas-api-e2e-inf-a-750972-9bf947464-z9l7z\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:14.484687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:14.484648 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:14.933039 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:39:14.933001 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e5d601_93ef_4424_b06c_bc5ca97943e5.slice/crio-57be42fd5853065685450efea418ce7a0a4721938048ec5c68fd3baa0e28657c WatchSource:0}: Error finding container 57be42fd5853065685450efea418ce7a0a4721938048ec5c68fd3baa0e28657c: Status 404 returned error can't find the container with id 57be42fd5853065685450efea418ce7a0a4721938048ec5c68fd3baa0e28657c Oct 29 23:39:14.934483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:14.934154 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z"] Oct 29 23:39:14.935485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:14.935459 5736 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:39:15.628701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:15.628594 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" event={"ID":"64e5d601-93ef-4424-b06c-bc5ca97943e5","Type":"ContainerStarted","Data":"e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5"} Oct 29 23:39:15.628919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:15.628699 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" event={"ID":"64e5d601-93ef-4424-b06c-bc5ca97943e5","Type":"ContainerStarted","Data":"57be42fd5853065685450efea418ce7a0a4721938048ec5c68fd3baa0e28657c"} Oct 29 23:39:15.628919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:15.628801 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:15.682243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:15.682190 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" podStartSLOduration=2.360064094 podStartE2EDuration="2.682175642s" totalImagesPullingTime="322.111548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:39:15.677800882 +0000 UTC m=+3352.793720489" watchObservedRunningTime="2026-10-29 23:39:15.682175642 +0000 UTC m=+3352.798095285" Oct 29 23:39:18.827818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:18.827781 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg"] Oct 29 23:39:21.642812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:21.642781 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:39:26.826769 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:39:26.826720 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:39:33.857709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:33.857648 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" podUID="4171d287-41c7-4c40-871b-2fd7a55d7362" containerName="main" containerID="cri-o://08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1" gracePeriod=45 Oct 29 23:39:34.149055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.149029 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:39:34.206334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.206297 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-dshm\") pod \"4171d287-41c7-4c40-871b-2fd7a55d7362\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " Oct 29 23:39:34.206627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.206356 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4171d287-41c7-4c40-871b-2fd7a55d7362-tls-certs\" (UniqueName: \"kubernetes.io/secret/4171d287-41c7-4c40-871b-2fd7a55d7362-tls-certs\") pod \"4171d287-41c7-4c40-871b-2fd7a55d7362\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " Oct 29 23:39:34.206627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.206528 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-home\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-home\") pod \"4171d287-41c7-4c40-871b-2fd7a55d7362\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " Oct 29 23:39:34.206846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.206816 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-home" pod "4171d287-41c7-4c40-871b-2fd7a55d7362" (UID: "4171d287-41c7-4c40-871b-2fd7a55d7362"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:39:34.206934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.206908 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-model-cache\") pod \"4171d287-41c7-4c40-871b-2fd7a55d7362\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " Oct 29 23:39:34.207019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.206984 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4171d287-41c7-4c40-871b-2fd7a55d7362-kube-api-access-k4xzm\" (UniqueName: \"kubernetes.io/projected/4171d287-41c7-4c40-871b-2fd7a55d7362-kube-api-access-k4xzm\") pod \"4171d287-41c7-4c40-871b-2fd7a55d7362\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " Oct 29 23:39:34.207132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.207114 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-tmp-dir\") pod \"4171d287-41c7-4c40-871b-2fd7a55d7362\" (UID: \"4171d287-41c7-4c40-871b-2fd7a55d7362\") " Oct 29 23:39:34.207984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.207957 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:34.208365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.208337 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-model-cache" pod "4171d287-41c7-4c40-871b-2fd7a55d7362" (UID: "4171d287-41c7-4c40-871b-2fd7a55d7362"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:39:34.208484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.208402 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-tmp-dir" pod "4171d287-41c7-4c40-871b-2fd7a55d7362" (UID: "4171d287-41c7-4c40-871b-2fd7a55d7362"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:39:34.208906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.208879 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-dshm" pod "4171d287-41c7-4c40-871b-2fd7a55d7362" (UID: "4171d287-41c7-4c40-871b-2fd7a55d7362"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:39:34.209057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.209033 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4171d287-41c7-4c40-871b-2fd7a55d7362-tls-certs" pod "4171d287-41c7-4c40-871b-2fd7a55d7362" (UID: "4171d287-41c7-4c40-871b-2fd7a55d7362"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:39:34.210644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.210611 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4171d287-41c7-4c40-871b-2fd7a55d7362-kube-api-access-k4xzm" pod "4171d287-41c7-4c40-871b-2fd7a55d7362" (UID: "4171d287-41c7-4c40-871b-2fd7a55d7362"). InnerVolumeSpecName "kube-api-access-k4xzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:39:34.309243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.309214 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:34.309367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.309247 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4xzm\" (UniqueName: \"kubernetes.io/projected/4171d287-41c7-4c40-871b-2fd7a55d7362-kube-api-access-k4xzm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:34.309367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.309262 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:34.309367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.309272 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4171d287-41c7-4c40-871b-2fd7a55d7362-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:34.309367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.309280 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4171d287-41c7-4c40-871b-2fd7a55d7362-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:34.818397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.818359 5736 generic.go:356] "Generic (PLEG): container finished" podID="4171d287-41c7-4c40-871b-2fd7a55d7362" containerID="08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1" exitCode=0 Oct 29 23:39:34.818615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.818410 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" Oct 29 23:39:34.818615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.818440 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" event={"ID":"4171d287-41c7-4c40-871b-2fd7a55d7362","Type":"ContainerDied","Data":"08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1"} Oct 29 23:39:34.818615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.818489 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" event={"ID":"4171d287-41c7-4c40-871b-2fd7a55d7362","Type":"ContainerDied","Data":"881cf5bf179bbc299b206da6fd6888a7f40f2bff597354687fe876ab929fe4ad"} Oct 29 23:39:34.818615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.818508 5736 scope.go:122] "RemoveContainer" containerID="08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1" Oct 29 23:39:34.831879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.831858 5736 scope.go:122] "RemoveContainer" containerID="08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1" Oct 29 23:39:34.832141 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:39:34.832117 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1\": container with ID starting with 08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1 not found: ID does not exist" containerID="08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1" Oct 29 23:39:34.832213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.832148 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1"} err="failed to get container status \"08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1\": rpc error: code = NotFound desc = could not find container \"08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1\": container with ID starting with 08ed9e7b68176a47b5cddab8a55281aa3c21aefefd0d0fee02699b92381b62b1 not found: ID does not exist" Oct 29 23:39:34.872074 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.872050 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg"] Oct 29 23:39:34.883497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:34.883473 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg"] Oct 29 23:39:35.008195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:35.008163 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4171d287-41c7-4c40-871b-2fd7a55d7362" path="/var/lib/kubelet/pods/4171d287-41c7-4c40-871b-2fd7a55d7362/volumes" Oct 29 23:39:35.067381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:35.067347 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-90f683/test-model-90f683-kserve-f485d9ff-ztspg" podUID="4171d287-41c7-4c40-871b-2fd7a55d7362" containerName="main" probeResult="failure" output="Get \"https://10.134.1.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:40.376698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.376656 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t"] Oct 29 23:39:40.756387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.756353 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx"] Oct 29 23:39:40.757436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.757408 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4171d287-41c7-4c40-871b-2fd7a55d7362" containerName="main" Oct 29 23:39:40.757436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.757432 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4171d287-41c7-4c40-871b-2fd7a55d7362" containerName="main" Oct 29 23:39:40.757645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.757634 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4171d287-41c7-4c40-871b-2fd7a55d7362" containerName="main" Oct 29 23:39:40.762491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.762467 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:40.777384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.777354 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:39:40.777497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.777360 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:39:40.782868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.782842 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx"] Oct 29 23:39:40.919968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.919934 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-90f683-crwsx\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:40.920221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.920192 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-90f683-crwsx\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:40.920394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:40.920379 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jq2t\" (UniqueName: \"kubernetes.io/projected/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-kube-api-access-4jq2t\") pod \"maas-api-revoke-keys-e2e-inf-a-90f683-crwsx\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:41.021822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.021736 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jq2t\" (UniqueName: \"kubernetes.io/projected/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-kube-api-access-4jq2t\") pod \"maas-api-revoke-keys-e2e-inf-a-90f683-crwsx\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:41.021966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.021903 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-90f683-crwsx\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:41.022121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.022103 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-90f683-crwsx\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:41.022815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.022794 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-90f683-crwsx\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:41.024509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.024486 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-90f683-crwsx\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:41.044625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.044602 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jq2t\" (UniqueName: \"kubernetes.io/projected/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-kube-api-access-4jq2t\") pod \"maas-api-revoke-keys-e2e-inf-a-90f683-crwsx\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:41.078299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.078276 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:41.400602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.397731 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx"] Oct 29 23:39:41.883984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.883947 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" event={"ID":"63976073-e6a3-40fb-81e3-9cffa4cc3f1a","Type":"ContainerStarted","Data":"56e1aacd6bd94c6993ac35f3b87c086dc044ca46782ca34ea94d3918991008e6"} Oct 29 23:39:41.884107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.884019 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" event={"ID":"63976073-e6a3-40fb-81e3-9cffa4cc3f1a","Type":"ContainerStarted","Data":"805863ed5510254fe0763177df201e100305fe356517569c26f918e2273cf3a0"} Oct 29 23:39:41.928109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:41.928063 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" podStartSLOduration=1.9280498499999998 podStartE2EDuration="1.92804985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:39:41.922953485 +0000 UTC m=+3379.038873091" watchObservedRunningTime="2026-10-29 23:39:41.92804985 +0000 UTC m=+3379.043969455" Oct 29 23:39:42.894343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:42.894258 5736 generic.go:356] "Generic (PLEG): container finished" podID="63976073-e6a3-40fb-81e3-9cffa4cc3f1a" containerID="56e1aacd6bd94c6993ac35f3b87c086dc044ca46782ca34ea94d3918991008e6" exitCode=0 Oct 29 23:39:42.894746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:42.894343 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" event={"ID":"63976073-e6a3-40fb-81e3-9cffa4cc3f1a","Type":"ContainerDied","Data":"56e1aacd6bd94c6993ac35f3b87c086dc044ca46782ca34ea94d3918991008e6"} Oct 29 23:39:44.080151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.080120 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:44.182523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.182453 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-xks-ca\") pod \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " Oct 29 23:39:44.182731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.182559 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-kube-api-access-4jq2t\" (UniqueName: \"kubernetes.io/projected/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-kube-api-access-4jq2t\") pod \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " Oct 29 23:39:44.182934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.182911 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-openshift-ca\") pod \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\" (UID: \"63976073-e6a3-40fb-81e3-9cffa4cc3f1a\") " Oct 29 23:39:44.183600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.183329 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-openshift-ca" pod "63976073-e6a3-40fb-81e3-9cffa4cc3f1a" (UID: "63976073-e6a3-40fb-81e3-9cffa4cc3f1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:39:44.184158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.184133 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:44.185162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.184998 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-xks-ca" pod "63976073-e6a3-40fb-81e3-9cffa4cc3f1a" (UID: "63976073-e6a3-40fb-81e3-9cffa4cc3f1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:39:44.185162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.185140 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-kube-api-access-4jq2t" pod "63976073-e6a3-40fb-81e3-9cffa4cc3f1a" (UID: "63976073-e6a3-40fb-81e3-9cffa4cc3f1a"). InnerVolumeSpecName "kube-api-access-4jq2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:39:44.286191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.286161 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jq2t\" (UniqueName: \"kubernetes.io/projected/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-kube-api-access-4jq2t\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:44.286191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.286189 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63976073-e6a3-40fb-81e3-9cffa4cc3f1a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:44.912869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.912840 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" Oct 29 23:39:44.913040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.912871 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx" event={"ID":"63976073-e6a3-40fb-81e3-9cffa4cc3f1a","Type":"ContainerDied","Data":"805863ed5510254fe0763177df201e100305fe356517569c26f918e2273cf3a0"} Oct 29 23:39:44.913040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:44.912916 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805863ed5510254fe0763177df201e100305fe356517569c26f918e2273cf3a0" Oct 29 23:39:49.331604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.324698 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.331604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.330601 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.331604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.331308 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.332251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.332000 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" podUID="9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" containerName="main" probeResult="failure" output="Get \"https://10.134.1.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.404605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.400706 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" podUID="edc14cbd-79a0-449f-a896-eda15ab761c6" containerName="main" probeResult="failure" output="Get \"https://10.134.1.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.404605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.401189 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.404605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.401506 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 23:39:49.404605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.401837 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 29 23:39:49.404605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.403238 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.404605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.403493 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 29 23:39:49.404605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.404201 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 29 23:39:49.404605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.404455 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 29 23:39:49.422609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.415799 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.441612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.440186 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" podUID="55389672-b516-4422-b514-3ca1355c361a" containerName="main" probeResult="failure" output="Get \"https://10.134.1.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.532649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.521812 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:49.686868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:49.685675 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:39:51.161820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:51.161783 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9"] Oct 29 23:39:55.427511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.427409 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" podUID="9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" containerName="main" containerID="cri-o://d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5" gracePeriod=45 Oct 29 23:39:55.743048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.743022 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:39:55.820606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.820560 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tls-certs\") pod \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " Oct 29 23:39:55.820857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.820834 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-dshm\") pod \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " Oct 29 23:39:55.821525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.821502 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-home\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-home\") pod \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " Oct 29 23:39:55.821840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.821809 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-home" pod "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" (UID: "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:39:55.821989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.821969 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-kube-api-access-k4llx\" (UniqueName: \"kubernetes.io/projected/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-kube-api-access-k4llx\") pod \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " Oct 29 23:39:55.822094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.822075 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tmp-dir\") pod \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " Oct 29 23:39:55.822181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.822115 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-model-cache\") pod \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\" (UID: \"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5\") " Oct 29 23:39:55.822703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.822675 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-model-cache" pod "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" (UID: "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:39:55.823039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.823017 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:55.823162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.823043 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:55.823162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.823137 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tmp-dir" pod "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" (UID: "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:39:55.824086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.824057 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tls-certs" pod "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" (UID: "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:39:55.824558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.824533 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-dshm" pod "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" (UID: "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:39:55.824700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.824638 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-kube-api-access-k4llx" pod "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" (UID: "9a5cbd9f-6e91-4839-abc9-cd84c9b533f5"). InnerVolumeSpecName "kube-api-access-k4llx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:39:55.924328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.924295 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:55.924328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.924327 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:55.924514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.924341 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4llx\" (UniqueName: \"kubernetes.io/projected/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-kube-api-access-k4llx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:55.924514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:55.924354 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:56.162654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.162618 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" podUID="476c2a23-7c51-4704-a62b-09a22abc1f2d" containerName="payload-processing" containerID="cri-o://b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb" gracePeriod=25 Oct 29 23:39:56.187116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.187087 5736 generic.go:356] "Generic (PLEG): container finished" podID="9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" containerID="d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5" exitCode=0 Oct 29 23:39:56.187236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.187125 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" Oct 29 23:39:56.187236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.187151 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" event={"ID":"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5","Type":"ContainerDied","Data":"d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5"} Oct 29 23:39:56.187236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.187186 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t" event={"ID":"9a5cbd9f-6e91-4839-abc9-cd84c9b533f5","Type":"ContainerDied","Data":"972bd55fb55a8d93d96349058c9f69c2f2a0cbce0968caa2b6c5cb27c69a3de7"} Oct 29 23:39:56.187236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.187205 5736 scope.go:122] "RemoveContainer" containerID="d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5" Oct 29 23:39:56.277366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.277326 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t"] Oct 29 23:39:56.283125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.283092 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9965bf/test-model-9965bf-kserve-68b58fdc65-8dh4t"] Oct 29 23:39:56.291413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.291387 5736 scope.go:122] "RemoveContainer" containerID="d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5" Oct 29 23:39:56.291806 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:39:56.291778 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5\": container with ID starting with d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5 not found: ID does not exist" containerID="d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5" Oct 29 23:39:56.291917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.291813 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5"} err="failed to get container status \"d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5\": rpc error: code = NotFound desc = could not find container \"d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5\": container with ID starting with d501b20517d9f4464871081bca961bf3061430c930939d9c154562b6a85f07f5 not found: ID does not exist" Oct 29 23:39:56.485243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.485217 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:39:56.553236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.553200 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476c2a23-7c51-4704-a62b-09a22abc1f2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476c2a23-7c51-4704-a62b-09a22abc1f2d-plugins-config-volume\") pod \"476c2a23-7c51-4704-a62b-09a22abc1f2d\" (UID: \"476c2a23-7c51-4704-a62b-09a22abc1f2d\") " Oct 29 23:39:56.553429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.553261 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476c2a23-7c51-4704-a62b-09a22abc1f2d-kube-api-access-jqlwd\" (UniqueName: \"kubernetes.io/projected/476c2a23-7c51-4704-a62b-09a22abc1f2d-kube-api-access-jqlwd\") pod \"476c2a23-7c51-4704-a62b-09a22abc1f2d\" (UID: \"476c2a23-7c51-4704-a62b-09a22abc1f2d\") " Oct 29 23:39:56.554200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.554159 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476c2a23-7c51-4704-a62b-09a22abc1f2d-plugins-config-volume" pod "476c2a23-7c51-4704-a62b-09a22abc1f2d" (UID: "476c2a23-7c51-4704-a62b-09a22abc1f2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:39:56.555992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.555949 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476c2a23-7c51-4704-a62b-09a22abc1f2d-kube-api-access-jqlwd" pod "476c2a23-7c51-4704-a62b-09a22abc1f2d" (UID: "476c2a23-7c51-4704-a62b-09a22abc1f2d"). InnerVolumeSpecName "kube-api-access-jqlwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:39:56.655529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.655486 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476c2a23-7c51-4704-a62b-09a22abc1f2d-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:56.655529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:56.655520 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqlwd\" (UniqueName: \"kubernetes.io/projected/476c2a23-7c51-4704-a62b-09a22abc1f2d-kube-api-access-jqlwd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:39:56.830630 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:39:56.830528 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:39:57.006977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.006932 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" path="/var/lib/kubelet/pods/9a5cbd9f-6e91-4839-abc9-cd84c9b533f5/volumes" Oct 29 23:39:57.198006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.197914 5736 generic.go:356] "Generic (PLEG): container finished" podID="476c2a23-7c51-4704-a62b-09a22abc1f2d" containerID="b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb" exitCode=0 Oct 29 23:39:57.198006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.197970 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" Oct 29 23:39:57.198006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.197995 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" event={"ID":"476c2a23-7c51-4704-a62b-09a22abc1f2d","Type":"ContainerDied","Data":"b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb"} Oct 29 23:39:57.198305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.198057 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9" event={"ID":"476c2a23-7c51-4704-a62b-09a22abc1f2d","Type":"ContainerDied","Data":"8a60fe27ce3f220a1ee742fc57609ffdc81716fe32eafcb4224c563b72949dd1"} Oct 29 23:39:57.198305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.198099 5736 scope.go:122] "RemoveContainer" containerID="b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb" Oct 29 23:39:57.211569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.211540 5736 scope.go:122] "RemoveContainer" containerID="b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb" Oct 29 23:39:57.212226 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:39:57.212185 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb\": container with ID starting with b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb not found: ID does not exist" containerID="b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb" Oct 29 23:39:57.212338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.212234 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb"} err="failed to get container status \"b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb\": rpc error: code = NotFound desc = could not find container \"b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb\": container with ID starting with b5c53805c6fc823c46f565b4beccc26b0550d3da69b6689a29479bc2d113abfb not found: ID does not exist" Oct 29 23:39:57.254127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.254087 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9"] Oct 29 23:39:57.265643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:57.265594 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f683-6bddf754fb-h6vm9"] Oct 29 23:39:59.007420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:39:59.007380 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476c2a23-7c51-4704-a62b-09a22abc1f2d" path="/var/lib/kubelet/pods/476c2a23-7c51-4704-a62b-09a22abc1f2d/volumes" Oct 29 23:40:00.232293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:00.232257 5736 generic.go:356] "Generic (PLEG): container finished" podID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerID="d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8" exitCode=0 Oct 29 23:40:00.232854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:00.232423 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" event={"ID":"5d529cec-33d9-44f2-b1a6-56dcba642a03","Type":"ContainerDied","Data":"d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8"} Oct 29 23:40:00.232990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:00.232969 5736 scope.go:122] "RemoveContainer" containerID="d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8" Oct 29 23:40:00.235821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:00.235793 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:00.235930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:00.235879 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:00.236005 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:00.235933 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:01.243338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:01.243300 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" event={"ID":"5d529cec-33d9-44f2-b1a6-56dcba642a03","Type":"ContainerStarted","Data":"63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b"} Oct 29 23:40:01.603614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:01.603518 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:40:01.984659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:01.984606 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:40:01.985753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:01.985727 5736 patch_prober.go:29] interesting pod/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 29 23:40:01.985824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:01.985785 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 29 23:40:02.984955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:02.984918 5736 patch_prober.go:29] interesting pod/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 29 23:40:02.985501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:02.984979 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 29 23:40:03.985747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:03.985709 5736 patch_prober.go:29] interesting pod/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 29 23:40:03.986171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:03.985773 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 29 23:40:04.992534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:04.992490 5736 patch_prober.go:29] interesting pod/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 29 23:40:04.993053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:04.992561 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 29 23:40:05.985158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:05.985116 5736 patch_prober.go:29] interesting pod/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 29 23:40:05.985340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:05.985207 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 29 23:40:06.985523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:06.985486 5736 patch_prober.go:29] interesting pod/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 29 23:40:06.986065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:06.985549 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 29 23:40:07.991315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:07.991281 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:40:07.992367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:07.992341 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:40:12.128501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.128466 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm"] Oct 29 23:40:12.129511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.129486 5736 cpu_manager.go:414] "containerMap: removing container" podUID="63976073-e6a3-40fb-81e3-9cffa4cc3f1a" containerName="revoke-keys" Oct 29 23:40:12.129511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.129510 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63976073-e6a3-40fb-81e3-9cffa4cc3f1a" containerName="revoke-keys" Oct 29 23:40:12.129746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.129545 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" containerName="main" Oct 29 23:40:12.129746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.129553 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" containerName="main" Oct 29 23:40:12.129746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.129619 5736 cpu_manager.go:414] "containerMap: removing container" podUID="476c2a23-7c51-4704-a62b-09a22abc1f2d" containerName="payload-processing" Oct 29 23:40:12.129746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.129628 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476c2a23-7c51-4704-a62b-09a22abc1f2d" containerName="payload-processing" Oct 29 23:40:12.129971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.129843 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="63976073-e6a3-40fb-81e3-9cffa4cc3f1a" containerName="revoke-keys" Oct 29 23:40:12.129971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.129866 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5cbd9f-6e91-4839-abc9-cd84c9b533f5" containerName="main" Oct 29 23:40:12.129971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.129885 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="476c2a23-7c51-4704-a62b-09a22abc1f2d" containerName="payload-processing" Oct 29 23:40:12.134878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.134852 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.151555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.151530 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-55c3fc-openshift-default-dockercfg-ds46m\"" Oct 29 23:40:12.171538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.171440 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm"] Oct 29 23:40:12.218126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.218086 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjd4j\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-kube-api-access-sjd4j\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.218372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.218349 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-data\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.218469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.218408 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-envoy\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.218642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.218622 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-token\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.218908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.218880 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-podinfo\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.218960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.218927 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-socket\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.219009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.218981 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istiod-ca-cert\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.219049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.219009 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-certs\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.219093 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.219050 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-credential-socket\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.323601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.322983 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-token\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.323601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.323237 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-podinfo\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.323601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.323269 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-socket\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.323601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.323322 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istiod-ca-cert\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.323601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.323348 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-certs\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.323601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.323387 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-credential-socket\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.323601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.323550 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjd4j\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-kube-api-access-sjd4j\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.324127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.323627 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-data\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.324127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.323679 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-envoy\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.356607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.337655 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istiod-ca-cert\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.387651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.362166 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-certs\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.387651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.362439 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-credential-socket\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.387651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.363048 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-data\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.387651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.382819 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-socket\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.441611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.428922 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjd4j\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-kube-api-access-sjd4j\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.481214 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.481172 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-token\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.526995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.526959 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-envoy\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.538604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.537990 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-podinfo\") pod \"e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:12.757905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:12.757868 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:13.203979 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:40:13.203932 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0946c19_9c6c_4fdc_bac2_cb943f88d8c2.slice/crio-bc712b4d2a27cb66dcd9c33189c116022970b7f28e8c8926274be4d3ba47c9ef WatchSource:0}: Error finding container bc712b4d2a27cb66dcd9c33189c116022970b7f28e8c8926274be4d3ba47c9ef: Status 404 returned error can't find the container with id bc712b4d2a27cb66dcd9c33189c116022970b7f28e8c8926274be4d3ba47c9ef Oct 29 23:40:13.210413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:13.210361 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:13.210737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:13.210693 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:13.211044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:13.211006 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:13.226591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:13.226538 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm"] Oct 29 23:40:13.530621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:13.530568 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" event={"ID":"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2","Type":"ContainerStarted","Data":"3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5"} Oct 29 23:40:13.530803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:13.530646 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" event={"ID":"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2","Type":"ContainerStarted","Data":"bc712b4d2a27cb66dcd9c33189c116022970b7f28e8c8926274be4d3ba47c9ef"} Oct 29 23:40:13.580132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:13.580062 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podStartSLOduration=1.5800419940000001 podStartE2EDuration="1.580041994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:13.576964874 +0000 UTC m=+3410.692884481" watchObservedRunningTime="2026-10-29 23:40:13.580041994 +0000 UTC m=+3410.695961603" Oct 29 23:40:13.762605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:13.760707 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:14.768015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:14.767973 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:15.553779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:15.553745 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:15.556894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:15.556865 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:40:26.827463 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:40:26.827423 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:40:29.105723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:29.105643 5736 scope.go:122] "RemoveContainer" containerID="03ace19a7380268de020e75d130208976b74fdb9d870e36539397f527b4bb1f9" Oct 29 23:40:33.827791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:33.827713 5736 generic.go:356] "Generic (PLEG): container finished" podID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerID="ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb" exitCode=0 Oct 29 23:40:33.828250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:33.827785 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" event={"ID":"afc2652b-8879-4e8c-8337-ec8faa807dd7","Type":"ContainerDied","Data":"ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb"} Oct 29 23:40:33.828250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:33.828177 5736 scope.go:122] "RemoveContainer" containerID="ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb" Oct 29 23:40:33.830279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:33.830245 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:33.830407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:33.830357 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:33.830492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:33.830408 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:34.865172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:34.865132 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" event={"ID":"afc2652b-8879-4e8c-8337-ec8faa807dd7","Type":"ContainerStarted","Data":"1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84"} Oct 29 23:40:35.572438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:35.572400 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:40:35.572677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:35.572653 5736 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 29 23:40:35.572745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:35.572709 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 29 23:40:36.573142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:36.573096 5736 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 29 23:40:36.573835 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:36.573799 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 29 23:40:37.573017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:37.572978 5736 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 29 23:40:37.573227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:37.573043 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 29 23:40:38.580669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:38.579835 5736 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 29 23:40:38.580669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:38.579899 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 29 23:40:39.614631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:39.592803 5736 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 29 23:40:39.617145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:39.617100 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 29 23:40:39.806599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:39.806552 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx"] Oct 29 23:40:39.817059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:39.817026 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-90f683-crwsx"] Oct 29 23:40:40.579013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:40.578967 5736 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Oct 29 23:40:40.579344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:40.579311 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Oct 29 23:40:41.008945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:41.008897 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63976073-e6a3-40fb-81e3-9cffa4cc3f1a" path="/var/lib/kubelet/pods/63976073-e6a3-40fb-81e3-9cffa4cc3f1a/volumes" Oct 29 23:40:41.578705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:41.578674 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:40:41.579017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:41.578988 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:40:41.579684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:41.579665 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:40:47.613804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:47.610741 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k"] Oct 29 23:40:47.613804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:47.611030 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" containerID="cri-o://181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25" gracePeriod=30 Oct 29 23:40:47.865723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:47.865630 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 29 23:40:48.067570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.067535 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr"] Oct 29 23:40:48.072279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.072256 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.088410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.088380 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:40:48.088554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.088459 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:40:48.096651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.094702 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr"] Oct 29 23:40:48.227158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.227120 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.227158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.227157 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.227470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.227376 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95h5\" (UniqueName: \"kubernetes.io/projected/2a6212de-bcdb-4cc6-8c9f-777193c2249f-kube-api-access-v95h5\") pod \"maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.329127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.329083 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.329342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.329138 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.329469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.329445 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95h5\" (UniqueName: \"kubernetes.io/projected/2a6212de-bcdb-4cc6-8c9f-777193c2249f-kube-api-access-v95h5\") pod \"maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.330094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.330072 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.332381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.332347 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.355085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.355054 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95h5\" (UniqueName: \"kubernetes.io/projected/2a6212de-bcdb-4cc6-8c9f-777193c2249f-kube-api-access-v95h5\") pod \"maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.388221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.388190 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:48.816295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:48.816260 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr"] Oct 29 23:40:48.817825 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:40:48.817796 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6212de_bcdb_4cc6_8c9f_777193c2249f.slice/crio-974890fb5ef021c45e286ee8d593fca030190f1cf6039dbfe5a804b8815d6e47 WatchSource:0}: Error finding container 974890fb5ef021c45e286ee8d593fca030190f1cf6039dbfe5a804b8815d6e47: Status 404 returned error can't find the container with id 974890fb5ef021c45e286ee8d593fca030190f1cf6039dbfe5a804b8815d6e47 Oct 29 23:40:49.560016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:49.559926 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" event={"ID":"2a6212de-bcdb-4cc6-8c9f-777193c2249f","Type":"ContainerStarted","Data":"7fddcdfdbe9d1aea84cb1f849cce2712555521d696700cbc872c4466fed23076"} Oct 29 23:40:49.560016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:49.559990 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" event={"ID":"2a6212de-bcdb-4cc6-8c9f-777193c2249f","Type":"ContainerStarted","Data":"974890fb5ef021c45e286ee8d593fca030190f1cf6039dbfe5a804b8815d6e47"} Oct 29 23:40:49.608424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:49.608373 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" podStartSLOduration=1.608359716 podStartE2EDuration="1.608359716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:49.601671585 +0000 UTC m=+3446.717591190" watchObservedRunningTime="2026-10-29 23:40:49.608359716 +0000 UTC m=+3446.724279322" Oct 29 23:40:49.983713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:49.983680 5736 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:40:49.984111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:49.983733 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:40:50.570398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:50.570360 5736 generic.go:356] "Generic (PLEG): container finished" podID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerID="d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230" exitCode=0 Oct 29 23:40:50.570651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:50.570433 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" event={"ID":"d69cbd5a-5121-4051-ab5c-7ef57eda66c4","Type":"ContainerDied","Data":"d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230"} Oct 29 23:40:50.571010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:50.570989 5736 scope.go:122] "RemoveContainer" containerID="d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230" Oct 29 23:40:50.572594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:50.572556 5736 generic.go:356] "Generic (PLEG): container finished" podID="2a6212de-bcdb-4cc6-8c9f-777193c2249f" containerID="7fddcdfdbe9d1aea84cb1f849cce2712555521d696700cbc872c4466fed23076" exitCode=0 Oct 29 23:40:50.572700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:50.572630 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" event={"ID":"2a6212de-bcdb-4cc6-8c9f-777193c2249f","Type":"ContainerDied","Data":"7fddcdfdbe9d1aea84cb1f849cce2712555521d696700cbc872c4466fed23076"} Oct 29 23:40:50.573538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:50.573494 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:50.573685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:50.573643 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:50.573757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:50.573718 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:40:51.589647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.589606 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" event={"ID":"d69cbd5a-5121-4051-ab5c-7ef57eda66c4","Type":"ContainerStarted","Data":"a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed"} Oct 29 23:40:51.811589 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.811551 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:51.922098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.922021 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-openshift-ca\") pod \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " Oct 29 23:40:51.922370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.922348 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6212de-bcdb-4cc6-8c9f-777193c2249f-kube-api-access-v95h5\" (UniqueName: \"kubernetes.io/projected/2a6212de-bcdb-4cc6-8c9f-777193c2249f-kube-api-access-v95h5\") pod \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " Oct 29 23:40:51.922428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.922394 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-xks-ca\") pod \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\" (UID: \"2a6212de-bcdb-4cc6-8c9f-777193c2249f\") " Oct 29 23:40:51.922428 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.922405 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-openshift-ca" pod "2a6212de-bcdb-4cc6-8c9f-777193c2249f" (UID: "2a6212de-bcdb-4cc6-8c9f-777193c2249f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:40:51.923208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.923183 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:51.924470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.924446 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6212de-bcdb-4cc6-8c9f-777193c2249f-kube-api-access-v95h5" pod "2a6212de-bcdb-4cc6-8c9f-777193c2249f" (UID: "2a6212de-bcdb-4cc6-8c9f-777193c2249f"). InnerVolumeSpecName "kube-api-access-v95h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:40:51.924560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.924471 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-xks-ca" pod "2a6212de-bcdb-4cc6-8c9f-777193c2249f" (UID: "2a6212de-bcdb-4cc6-8c9f-777193c2249f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:40:51.997857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.997825 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:40:51.998868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.998841 5736 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Oct 29 23:40:51.998984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:51.998888 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Oct 29 23:40:52.025106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:52.025070 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v95h5\" (UniqueName: \"kubernetes.io/projected/2a6212de-bcdb-4cc6-8c9f-777193c2249f-kube-api-access-v95h5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:52.025106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:52.025099 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6212de-bcdb-4cc6-8c9f-777193c2249f-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:52.599456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:52.599430 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" Oct 29 23:40:52.599988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:52.599462 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr" event={"ID":"2a6212de-bcdb-4cc6-8c9f-777193c2249f","Type":"ContainerDied","Data":"974890fb5ef021c45e286ee8d593fca030190f1cf6039dbfe5a804b8815d6e47"} Oct 29 23:40:52.599988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:52.599509 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="974890fb5ef021c45e286ee8d593fca030190f1cf6039dbfe5a804b8815d6e47" Oct 29 23:40:52.982339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:52.982314 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:40:52.998307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:52.998277 5736 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Oct 29 23:40:52.998426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:52.998324 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Oct 29 23:40:53.069000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.068965 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw"] Oct 29 23:40:53.069830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.069809 5736 cpu_manager.go:414] "containerMap: removing container" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" Oct 29 23:40:53.069893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.069833 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" Oct 29 23:40:53.069893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.069852 5736 cpu_manager.go:414] "containerMap: removing container" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" Oct 29 23:40:53.069893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.069858 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" Oct 29 23:40:53.069893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.069865 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2a6212de-bcdb-4cc6-8c9f-777193c2249f" containerName="revoke-keys" Oct 29 23:40:53.069893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.069870 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6212de-bcdb-4cc6-8c9f-777193c2249f" containerName="revoke-keys" Oct 29 23:40:53.070066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.070010 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" Oct 29 23:40:53.070066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.070019 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerName="istio-proxy" Oct 29 23:40:53.070066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.070028 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6212de-bcdb-4cc6-8c9f-777193c2249f" containerName="revoke-keys" Oct 29 23:40:53.070905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.070881 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-podinfo\") pod \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " Oct 29 23:40:53.070967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.070919 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-credential-socket\") pod \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " Oct 29 23:40:53.071004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.070970 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-socket\") pod \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " Oct 29 23:40:53.071261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.071233 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-credential-socket" pod "60d0e8aa-75af-49c5-bc60-043e2d3a16df" (UID: "60d0e8aa-75af-49c5-bc60-043e2d3a16df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:40:53.071319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.071254 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-socket" pod "60d0e8aa-75af-49c5-bc60-043e2d3a16df" (UID: "60d0e8aa-75af-49c5-bc60-043e2d3a16df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:40:53.071319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.071295 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-envoy\") pod \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " Oct 29 23:40:53.071408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.071385 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istiod-ca-cert\") pod \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " Oct 29 23:40:53.071457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.071439 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-token\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-token\") pod \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " Oct 29 23:40:53.071553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.071529 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-kube-api-access-g66jk\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-kube-api-access-g66jk\") pod \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " Oct 29 23:40:53.071803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.071778 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-certs\") pod \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " Oct 29 23:40:53.071917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.071854 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-data\") pod \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\" (UID: \"60d0e8aa-75af-49c5-bc60-043e2d3a16df\") " Oct 29 23:40:53.072423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.072201 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istiod-ca-cert" pod "60d0e8aa-75af-49c5-bc60-043e2d3a16df" (UID: "60d0e8aa-75af-49c5-bc60-043e2d3a16df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:40:53.072423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.072386 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-data" pod "60d0e8aa-75af-49c5-bc60-043e2d3a16df" (UID: "60d0e8aa-75af-49c5-bc60-043e2d3a16df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:40:53.072685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.072430 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-certs" pod "60d0e8aa-75af-49c5-bc60-043e2d3a16df" (UID: "60d0e8aa-75af-49c5-bc60-043e2d3a16df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:40:53.072926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.072889 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:53.072926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.072922 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:53.073069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.072937 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:53.073069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.072952 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:53.073069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.072965 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:53.076051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.074179 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-envoy" pod "60d0e8aa-75af-49c5-bc60-043e2d3a16df" (UID: "60d0e8aa-75af-49c5-bc60-043e2d3a16df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:40:53.087641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.084050 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw"] Oct 29 23:40:53.087641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.084222 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.087641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.087557 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-kube-api-access-g66jk" pod "60d0e8aa-75af-49c5-bc60-043e2d3a16df" (UID: "60d0e8aa-75af-49c5-bc60-043e2d3a16df"). InnerVolumeSpecName "kube-api-access-g66jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:40:53.088019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.087981 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-token" pod "60d0e8aa-75af-49c5-bc60-043e2d3a16df" (UID: "60d0e8aa-75af-49c5-bc60-043e2d3a16df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:40:53.088625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.088561 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-podinfo" pod "60d0e8aa-75af-49c5-bc60-043e2d3a16df" (UID: "60d0e8aa-75af-49c5-bc60-043e2d3a16df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:40:53.100470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.100415 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-37498a-serving-cert\"" Oct 29 23:40:53.174417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.174352 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv9g7\" (UniqueName: \"kubernetes.io/projected/234594a5-0c76-43e8-b46e-c296fcb6d9ea-kube-api-access-zv9g7\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.174550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.174415 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-maas-api-tls\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.174617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.174550 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234594a5-0c76-43e8-b46e-c296fcb6d9ea-openshift-service-ca\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.174795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.174769 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-metrics-certs\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.174950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.174937 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:53.174991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.174953 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:53.174991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.174962 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:53.174991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.174972 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g66jk\" (UniqueName: \"kubernetes.io/projected/60d0e8aa-75af-49c5-bc60-043e2d3a16df-kube-api-access-g66jk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:40:53.276972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.276926 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9g7\" (UniqueName: \"kubernetes.io/projected/234594a5-0c76-43e8-b46e-c296fcb6d9ea-kube-api-access-zv9g7\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.277155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.277018 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-maas-api-tls\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.277226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.277157 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234594a5-0c76-43e8-b46e-c296fcb6d9ea-openshift-service-ca\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.277364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.277344 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-metrics-certs\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.277897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.277873 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234594a5-0c76-43e8-b46e-c296fcb6d9ea-openshift-service-ca\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.279549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.279519 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-maas-api-tls\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.279739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.279718 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-metrics-certs\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.300225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.300199 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9g7\" (UniqueName: \"kubernetes.io/projected/234594a5-0c76-43e8-b46e-c296fcb6d9ea-kube-api-access-zv9g7\") pod \"maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.466841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.466797 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:53.617755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.615725 5736 generic.go:356] "Generic (PLEG): container finished" podID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" containerID="181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25" exitCode=0 Oct 29 23:40:53.617755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.615915 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" event={"ID":"60d0e8aa-75af-49c5-bc60-043e2d3a16df","Type":"ContainerDied","Data":"181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25"} Oct 29 23:40:53.617755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.615977 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" event={"ID":"60d0e8aa-75af-49c5-bc60-043e2d3a16df","Type":"ContainerDied","Data":"d4cb10bd9f6d50992b23db33c7c6ea915a43c05797ddf67e96f0c5624a61ac58"} Oct 29 23:40:53.617755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.615999 5736 scope.go:122] "RemoveContainer" containerID="181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25" Oct 29 23:40:53.617755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.616154 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k" Oct 29 23:40:53.630400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.630374 5736 scope.go:122] "RemoveContainer" containerID="0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1" Oct 29 23:40:53.670470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.670445 5736 scope.go:122] "RemoveContainer" containerID="181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25" Oct 29 23:40:53.670976 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:40:53.670844 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25\": container with ID starting with 181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25 not found: ID does not exist" containerID="181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25" Oct 29 23:40:53.670976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.670884 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25"} err="failed to get container status \"181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25\": rpc error: code = NotFound desc = could not find container \"181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25\": container with ID starting with 181149d2fac25f5b10433552198a1196932a8f147c6026d0587f80f4be65bd25 not found: ID does not exist" Oct 29 23:40:53.670976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.670905 5736 scope.go:122] "RemoveContainer" containerID="0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1" Oct 29 23:40:53.671240 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:40:53.671209 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1\": container with ID starting with 0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1 not found: ID does not exist" containerID="0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1" Oct 29 23:40:53.671307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.671248 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1"} err="failed to get container status \"0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1\": rpc error: code = NotFound desc = could not find container \"0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1\": container with ID starting with 0e150c5241bc301920bb9de3954fb3219c92bf9f82745f2c4564d0b248f8aef1 not found: ID does not exist" Oct 29 23:40:53.685472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.685442 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k"] Oct 29 23:40:53.695953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.695931 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-p946k"] Oct 29 23:40:53.794996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.794922 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw"] Oct 29 23:40:53.798906 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:40:53.798872 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234594a5_0c76_43e8_b46e_c296fcb6d9ea.slice/crio-23194217f946a0b923ae3484a1cd9fe3a0709a3bc2c22c5e00c63e825670851e WatchSource:0}: Error finding container 23194217f946a0b923ae3484a1cd9fe3a0709a3bc2c22c5e00c63e825670851e: Status 404 returned error can't find the container with id 23194217f946a0b923ae3484a1cd9fe3a0709a3bc2c22c5e00c63e825670851e Oct 29 23:40:53.998549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.998518 5736 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Oct 29 23:40:53.998758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:53.998569 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Oct 29 23:40:54.629786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:54.629746 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" event={"ID":"234594a5-0c76-43e8-b46e-c296fcb6d9ea","Type":"ContainerStarted","Data":"68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3"} Oct 29 23:40:54.630174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:54.629811 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" event={"ID":"234594a5-0c76-43e8-b46e-c296fcb6d9ea","Type":"ContainerStarted","Data":"23194217f946a0b923ae3484a1cd9fe3a0709a3bc2c22c5e00c63e825670851e"} Oct 29 23:40:54.630174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:54.629845 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:40:54.998083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:54.998050 5736 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Oct 29 23:40:54.998269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:54.998112 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Oct 29 23:40:55.007027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:55.006997 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d0e8aa-75af-49c5-bc60-043e2d3a16df" path="/var/lib/kubelet/pods/60d0e8aa-75af-49c5-bc60-043e2d3a16df/volumes" Oct 29 23:40:55.998713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:55.998679 5736 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Oct 29 23:40:55.999138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:55.998734 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Oct 29 23:40:56.166696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:56.166657 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:40:56.826560 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:40:56.826523 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:40:56.998795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:56.998762 5736 patch_prober.go:29] interesting pod/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Oct 29 23:40:56.999206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:56.998810 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Oct 29 23:40:58.003790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:58.003754 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:40:58.004988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:58.004962 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:40:58.068210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:40:58.068163 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" podStartSLOduration=32.690244699 podStartE2EDuration="33.068150027s" totalImagesPullingTime="377.905328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:54.677222985 +0000 UTC m=+3451.793142590" watchObservedRunningTime="2026-10-29 23:40:58.068150027 +0000 UTC m=+3455.184069679" Oct 29 23:41:00.741603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:41:00.727766 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:41:26.826029 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:41:26.825978 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:41:51.032941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:41:51.032905 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd"] Oct 29 23:41:51.045152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:41:51.045120 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cecc986f-7z4bd"] Oct 29 23:41:53.006715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:41:53.006670 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db92548e-5f08-4922-b4a8-88282d3557ed" path="/var/lib/kubelet/pods/db92548e-5f08-4922-b4a8-88282d3557ed/volumes" Oct 29 23:41:53.080329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:41:53.080293 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g"] Oct 29 23:41:53.092258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:41:53.092216 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e98629-5kq2g"] Oct 29 23:41:55.006805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:41:55.006771 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f306b1-59ee-4f6b-951f-9f5b85d54ccc" path="/var/lib/kubelet/pods/c4f306b1-59ee-4f6b-951f-9f5b85d54ccc/volumes" Oct 29 23:41:56.827312 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:41:56.827271 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:42:02.046593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:02.046496 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt"] Oct 29 23:42:02.055914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:02.055864 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc"] Oct 29 23:42:02.060452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:02.060419 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5babbf-ns7dt"] Oct 29 23:42:02.065165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:02.065134 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc58fbf-v2xfc"] Oct 29 23:42:03.035662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:03.035564 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107c1179-356a-4afb-9fe5-6b0dad4a2868" path="/var/lib/kubelet/pods/107c1179-356a-4afb-9fe5-6b0dad4a2868/volumes" Oct 29 23:42:03.036301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:03.036277 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ee1d8d-96b9-4fe2-9371-dac4613d4f64" path="/var/lib/kubelet/pods/f1ee1d8d-96b9-4fe2-9371-dac4613d4f64/volumes" Oct 29 23:42:04.036039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:04.035956 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6"] Oct 29 23:42:04.050223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:04.050185 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94559fe6-r27h6"] Oct 29 23:42:05.006751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:05.006718 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97427e0e-39e7-4ff2-926e-2357a1ac7b05" path="/var/lib/kubelet/pods/97427e0e-39e7-4ff2-926e-2357a1ac7b05/volumes" Oct 29 23:42:08.617545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:08.617503 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr"] Oct 29 23:42:08.629055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:08.629023 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9965bf-qr5xr"] Oct 29 23:42:09.006468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:09.006433 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6212de-bcdb-4cc6-8c9f-777193c2249f" path="/var/lib/kubelet/pods/2a6212de-bcdb-4cc6-8c9f-777193c2249f/volumes" Oct 29 23:42:14.031339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:14.031291 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r"] Oct 29 23:42:14.042676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:14.042649 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f875a1-qbp6r"] Oct 29 23:42:15.019601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.017953 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp"] Oct 29 23:42:15.099665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.099232 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trccd\" (UniqueName: \"kubernetes.io/projected/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-kube-api-access-trccd\") pod \"payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp\" (UID: \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:15.100804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.100684 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp\" (UID: \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:15.101166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.100384 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:15.168853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.161525 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-825472-dockercfg-mt4lv\"" Oct 29 23:42:15.168853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.161716 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-825472\"" Oct 29 23:42:15.178652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.178619 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd3d048-8f0e-4217-98b4-f7400ab490f7" path="/var/lib/kubelet/pods/dcd3d048-8f0e-4217-98b4-f7400ab490f7/volumes" Oct 29 23:42:15.232757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.232717 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp"] Oct 29 23:42:15.242199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.240727 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trccd\" (UniqueName: \"kubernetes.io/projected/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-kube-api-access-trccd\") pod \"payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp\" (UID: \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:15.242199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.240775 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp\" (UID: \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:15.242199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.241875 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp\" (UID: \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:15.267456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.267427 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trccd\" (UniqueName: \"kubernetes.io/projected/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-kube-api-access-trccd\") pod \"payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp\" (UID: \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:15.482364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.482329 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:15.803925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:15.803894 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp"] Oct 29 23:42:15.806155 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:42:15.806126 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fed7e2_90a5_41e2_a1e0_22c694c3ef8b.slice/crio-70dfc01ad4f0224433aac8b375f54e88c03bdc5de5ef2764b2a784c99b55318f WatchSource:0}: Error finding container 70dfc01ad4f0224433aac8b375f54e88c03bdc5de5ef2764b2a784c99b55318f: Status 404 returned error can't find the container with id 70dfc01ad4f0224433aac8b375f54e88c03bdc5de5ef2764b2a784c99b55318f Oct 29 23:42:16.094742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.094658 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9"] Oct 29 23:42:16.098618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.094892 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" containerID="cri-o://ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b" gracePeriod=30 Oct 29 23:42:16.115096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.114741 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" event={"ID":"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b","Type":"ContainerStarted","Data":"f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450"} Oct 29 23:42:16.115096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.114804 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" event={"ID":"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b","Type":"ContainerStarted","Data":"70dfc01ad4f0224433aac8b375f54e88c03bdc5de5ef2764b2a784c99b55318f"} Oct 29 23:42:16.118600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.116558 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:16.138655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.138621 5736 patch_prober.go:29] interesting pod/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:42:16.138889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.138854 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:42:16.820017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.819965 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" podStartSLOduration=2.819949278 podStartE2EDuration="2.819949278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:42:16.261040499 +0000 UTC m=+3533.376960108" watchObservedRunningTime="2026-10-29 23:42:16.819949278 +0000 UTC m=+3533.935868928" Oct 29 23:42:16.821440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.821410 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls"] Oct 29 23:42:16.825672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.825647 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:16.854332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.854303 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:42:16.854481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.854344 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:42:16.860060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:16.860035 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls"] Oct 29 23:42:17.000793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.000757 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.000992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.000885 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qkf5\" (UniqueName: \"kubernetes.io/projected/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-kube-api-access-8qkf5\") pod \"maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.001077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.001024 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.103019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.102912 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.103283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.103260 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.103516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.103487 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qkf5\" (UniqueName: \"kubernetes.io/projected/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-kube-api-access-8qkf5\") pod \"maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.104057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.104034 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.105949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.105928 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.127101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.127075 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qkf5\" (UniqueName: \"kubernetes.io/projected/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-kube-api-access-8qkf5\") pod \"maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.139842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.139814 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:17.452466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:17.452436 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls"] Oct 29 23:42:17.454482 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:42:17.454450 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53b91f3_0411_4d0d_a9c4_9f1e5034dcd7.slice/crio-d6089763dbef6ba782ffa209e7ddc2e58fafa460997708d25b40f509328c2a0e WatchSource:0}: Error finding container d6089763dbef6ba782ffa209e7ddc2e58fafa460997708d25b40f509328c2a0e: Status 404 returned error can't find the container with id d6089763dbef6ba782ffa209e7ddc2e58fafa460997708d25b40f509328c2a0e Oct 29 23:42:18.135436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:18.135356 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" event={"ID":"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7","Type":"ContainerStarted","Data":"ee639e4dfca1fc57ee8e9456df87c0d84a41ab1b331dd4dc08390e6d0159af67"} Oct 29 23:42:18.135436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:18.135424 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" event={"ID":"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7","Type":"ContainerStarted","Data":"d6089763dbef6ba782ffa209e7ddc2e58fafa460997708d25b40f509328c2a0e"} Oct 29 23:42:18.177338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:18.177288 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" podStartSLOduration=2.177273895 podStartE2EDuration="2.177273895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:42:18.174746111 +0000 UTC m=+3535.290665717" watchObservedRunningTime="2026-10-29 23:42:18.177273895 +0000 UTC m=+3535.293193502" Oct 29 23:42:19.147136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:19.147094 5736 generic.go:356] "Generic (PLEG): container finished" podID="c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7" containerID="ee639e4dfca1fc57ee8e9456df87c0d84a41ab1b331dd4dc08390e6d0159af67" exitCode=0 Oct 29 23:42:19.147543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:19.147162 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" event={"ID":"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7","Type":"ContainerDied","Data":"ee639e4dfca1fc57ee8e9456df87c0d84a41ab1b331dd4dc08390e6d0159af67"} Oct 29 23:42:20.354587 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.354548 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:20.464221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.464181 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-xks-ca\") pod \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " Oct 29 23:42:20.464647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.464617 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-openshift-ca\") pod \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " Oct 29 23:42:20.464784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.464717 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-kube-api-access-8qkf5\" (UniqueName: \"kubernetes.io/projected/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-kube-api-access-8qkf5\") pod \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\" (UID: \"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7\") " Oct 29 23:42:20.465229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.464997 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-openshift-ca" pod "c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7" (UID: "c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:42:20.465871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.465840 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:20.467707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.467670 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-kube-api-access-8qkf5" pod "c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7" (UID: "c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7"). InnerVolumeSpecName "kube-api-access-8qkf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:42:20.467827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.467699 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-xks-ca" pod "c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7" (UID: "c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:42:20.570805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.570756 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qkf5\" (UniqueName: \"kubernetes.io/projected/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-kube-api-access-8qkf5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:20.570805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:20.570804 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.173170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.173139 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" Oct 29 23:42:21.173170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.173158 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls" event={"ID":"c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7","Type":"ContainerDied","Data":"d6089763dbef6ba782ffa209e7ddc2e58fafa460997708d25b40f509328c2a0e"} Oct 29 23:42:21.173170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.173241 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6089763dbef6ba782ffa209e7ddc2e58fafa460997708d25b40f509328c2a0e" Oct 29 23:42:21.403621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.402832 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:42:21.466405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.466373 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd"] Oct 29 23:42:21.467557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.467526 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" Oct 29 23:42:21.467557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.467557 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" Oct 29 23:42:21.467779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.467630 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7" containerName="revoke-keys" Oct 29 23:42:21.467779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.467639 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7" containerName="revoke-keys" Oct 29 23:42:21.467923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.467903 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" Oct 29 23:42:21.467964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.467926 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7" containerName="revoke-keys" Oct 29 23:42:21.467964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.467955 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" Oct 29 23:42:21.471537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.471510 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.486043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486013 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-envoy\") pod \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " Oct 29 23:42:21.486195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486091 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istiod-ca-cert\") pod \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " Oct 29 23:42:21.486275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486219 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-data\") pod \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " Oct 29 23:42:21.486275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486264 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-certs\") pod \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " Oct 29 23:42:21.486394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486304 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-podinfo\") pod \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " Oct 29 23:42:21.486483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486460 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-kube-api-access-9bddf\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-kube-api-access-9bddf\") pod \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " Oct 29 23:42:21.487157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486572 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-credential-socket\") pod \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " Oct 29 23:42:21.487157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486633 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-socket\") pod \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " Oct 29 23:42:21.487157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486678 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-token\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-token\") pod \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\" (UID: \"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e\") " Oct 29 23:42:21.487157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.486930 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istiod-ca-cert" pod "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" (UID: "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:42:21.487653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.487627 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.488270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.488246 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-envoy" pod "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" (UID: "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:42:21.488496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.488435 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-credential-socket" pod "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" (UID: "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:42:21.488620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.488564 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-certs" pod "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" (UID: "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:42:21.488732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.488712 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-socket" pod "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" (UID: "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:42:21.489344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.489319 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-kube-api-access-9bddf" pod "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" (UID: "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e"). InnerVolumeSpecName "kube-api-access-9bddf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:42:21.490190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.490171 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-token" pod "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" (UID: "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:42:21.490660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.490606 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-data" pod "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" (UID: "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:42:21.490886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.490866 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-podinfo" pod "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" (UID: "ecabf71e-aeb5-4a40-a591-1dfbb4f0468e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:42:21.499711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.499685 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd"] Oct 29 23:42:21.589221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.589172 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-data\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.589382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.589235 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-credential-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.589382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.589366 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bblbd\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-kube-api-access-bblbd\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.589534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.589519 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-token\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.589614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.589554 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-certs\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.589725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.589704 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-podinfo\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.589795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.589743 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.589862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.589849 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istiod-ca-cert\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.589936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.589913 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-envoy\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.590106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.590091 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.590184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.590111 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.590184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.590125 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.590184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.590140 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.590184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.590154 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.590184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.590167 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.590184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.590180 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.590388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.590194 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bddf\" (UniqueName: \"kubernetes.io/projected/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e-kube-api-access-9bddf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:21.692020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.691991 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bblbd\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-kube-api-access-bblbd\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692107 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-token\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692133 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-certs\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692195 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-podinfo\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692218 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692267 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istiod-ca-cert\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692294 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-envoy\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692370 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-data\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692406 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-credential-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692703 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-certs\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692790 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-data\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.692909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.692882 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-credential-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.693046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.693023 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-socket\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.693119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.693103 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istiod-ca-cert\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.694614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.694594 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-envoy\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.694888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.694867 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-podinfo\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.715619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.715575 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-token\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.715832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.715811 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bblbd\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-kube-api-access-bblbd\") pod \"e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:21.792573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:21.792523 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:22.122695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.122658 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd"] Oct 29 23:42:22.123932 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:42:22.123908 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4f22e2_6ee6_4b14_a672_e44a57fb5270.slice/crio-82667c909b8e8ca5fd7276909657347c67440f278c6fb2fd96f49cf7ed5ad69c WatchSource:0}: Error finding container 82667c909b8e8ca5fd7276909657347c67440f278c6fb2fd96f49cf7ed5ad69c: Status 404 returned error can't find the container with id 82667c909b8e8ca5fd7276909657347c67440f278c6fb2fd96f49cf7ed5ad69c Oct 29 23:42:22.127195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.127164 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:42:22.127275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.127229 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:42:22.127275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.127261 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:42:22.184157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.184134 5736 generic.go:356] "Generic (PLEG): container finished" podID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerID="ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b" exitCode=0 Oct 29 23:42:22.184267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.184179 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" Oct 29 23:42:22.184267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.184217 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" event={"ID":"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e","Type":"ContainerDied","Data":"ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b"} Oct 29 23:42:22.184356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.184276 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9" event={"ID":"ecabf71e-aeb5-4a40-a591-1dfbb4f0468e","Type":"ContainerDied","Data":"320422ec1f390a1a800998ae5de7ae4419bdc495ec15bcd0a512b49edc6f1717"} Oct 29 23:42:22.184356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.184305 5736 scope.go:122] "RemoveContainer" containerID="ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b" Oct 29 23:42:22.186077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.186048 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" event={"ID":"ec4f22e2-6ee6-4b14-a672-e44a57fb5270","Type":"ContainerStarted","Data":"82667c909b8e8ca5fd7276909657347c67440f278c6fb2fd96f49cf7ed5ad69c"} Oct 29 23:42:22.198151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.198130 5736 scope.go:122] "RemoveContainer" containerID="9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd" Oct 29 23:42:22.210321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.210297 5736 scope.go:122] "RemoveContainer" containerID="ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b" Oct 29 23:42:22.210634 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:42:22.210607 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b\": container with ID starting with ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b not found: ID does not exist" containerID="ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b" Oct 29 23:42:22.210726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.210642 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b"} err="failed to get container status \"ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b\": rpc error: code = NotFound desc = could not find container \"ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b\": container with ID starting with ab32ed3c82f2f6588024ed3dc022b99d3448496a66386010c0db42d25d4cfe4b not found: ID does not exist" Oct 29 23:42:22.210726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.210661 5736 scope.go:122] "RemoveContainer" containerID="9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd" Oct 29 23:42:22.210932 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:42:22.210912 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd\": container with ID starting with 9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd not found: ID does not exist" containerID="9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd" Oct 29 23:42:22.210976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.210939 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd"} err="failed to get container status \"9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd\": rpc error: code = NotFound desc = could not find container \"9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd\": container with ID starting with 9a5cdf060fab5585b54b15c55db7ee6990d2ab0b0fb047b65d5a4a08be05e1dd not found: ID does not exist" Oct 29 23:42:22.239446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.239421 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9"] Oct 29 23:42:22.252523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:22.252493 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9965bf-openshift-default-659d7fbb47-vp7p9"] Oct 29 23:42:23.016766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:23.016725 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" path="/var/lib/kubelet/pods/ecabf71e-aeb5-4a40-a591-1dfbb4f0468e/volumes" Oct 29 23:42:23.201417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:23.201356 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" event={"ID":"ec4f22e2-6ee6-4b14-a672-e44a57fb5270","Type":"ContainerStarted","Data":"ef97e707d4f40077988eb36feb135293cd9677b217edd67da13073d2ac750538"} Oct 29 23:42:23.277731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:23.277231 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" podStartSLOduration=64.277208111 podStartE2EDuration="1m4.277208111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:42:23.260119845 +0000 UTC m=+3540.376039454" watchObservedRunningTime="2026-10-29 23:42:23.277208111 +0000 UTC m=+3540.393127721" Oct 29 23:42:23.792763 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:23.792732 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:23.794838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:23.794810 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Oct 29 23:42:23.794983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:23.794895 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Oct 29 23:42:24.798603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:24.798021 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Oct 29 23:42:24.798603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:24.798105 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Oct 29 23:42:25.067654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:25.066728 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m"] Oct 29 23:42:25.112605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:25.112393 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e256980-chf8m"] Oct 29 23:42:25.793745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:25.793708 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Oct 29 23:42:25.793932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:25.793768 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Oct 29 23:42:26.792880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:26.792845 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Oct 29 23:42:26.793290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:26.792895 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Oct 29 23:42:26.826417 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:42:26.826377 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:42:27.006755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:27.006674 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58937dda-a893-4379-bf97-4efc3c4d3529" path="/var/lib/kubelet/pods/58937dda-a893-4379-bf97-4efc3c4d3529/volumes" Oct 29 23:42:27.793420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:27.793378 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Oct 29 23:42:27.793860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:27.793442 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Oct 29 23:42:28.204707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:28.190755 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:42:28.793139 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:28.793108 5736 patch_prober.go:29] interesting pod/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Oct 29 23:42:28.793326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:28.793167 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Oct 29 23:42:29.798442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:29.798410 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:29.798905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:29.798881 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:29.799487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:29.799470 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:42:37.030808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:37.030767 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk"] Oct 29 23:42:37.042102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:37.042073 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48be9aa-tvpqk"] Oct 29 23:42:39.006900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:39.006857 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593ca3e3-1b25-47b5-9117-9e23838310f8" path="/var/lib/kubelet/pods/593ca3e3-1b25-47b5-9117-9e23838310f8/volumes" Oct 29 23:42:54.721272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:54.721240 5736 generic.go:356] "Generic (PLEG): container finished" podID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerID="3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5" exitCode=0 Oct 29 23:42:54.721742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:54.721283 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" event={"ID":"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2","Type":"ContainerDied","Data":"3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5"} Oct 29 23:42:54.721742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:54.721651 5736 scope.go:122] "RemoveContainer" containerID="3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5" Oct 29 23:42:54.723800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:54.723770 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:42:54.723894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:54.723840 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:42:54.723894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:54.723874 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:42:55.732466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:55.732424 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" event={"ID":"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2","Type":"ContainerStarted","Data":"6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899"} Oct 29 23:42:55.761281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:55.761249 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:42:55.761442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:55.761369 5736 patch_prober.go:29] interesting pod/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" start-of-body= Oct 29 23:42:55.761442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:55.761409 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" Oct 29 23:42:56.760999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:56.760962 5736 patch_prober.go:29] interesting pod/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" start-of-body= Oct 29 23:42:56.761557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:56.761026 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" Oct 29 23:42:56.859499 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:42:56.859458 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:42:57.761487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:57.761453 5736 patch_prober.go:29] interesting pod/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" start-of-body= Oct 29 23:42:57.761993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:57.761517 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" Oct 29 23:42:58.760938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:58.760901 5736 patch_prober.go:29] interesting pod/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" start-of-body= Oct 29 23:42:58.761133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:58.760952 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" Oct 29 23:42:58.767343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:58.767312 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv"] Oct 29 23:42:58.767842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:58.767515 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" podUID="d8dbbcec-24d4-4f48-a086-c62bc4e1d379" containerName="maas-api" containerID="cri-o://40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18" gracePeriod=30 Oct 29 23:42:59.779704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.779668 5736 patch_prober.go:29] interesting pod/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" start-of-body= Oct 29 23:42:59.783268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.783228 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" Oct 29 23:42:59.800778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.800747 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:42:59.854304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.853639 5736 generic.go:356] "Generic (PLEG): container finished" podID="d8dbbcec-24d4-4f48-a086-c62bc4e1d379" containerID="40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18" exitCode=0 Oct 29 23:42:59.854304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.853810 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" event={"ID":"d8dbbcec-24d4-4f48-a086-c62bc4e1d379","Type":"ContainerDied","Data":"40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18"} Oct 29 23:42:59.854304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.853859 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" event={"ID":"d8dbbcec-24d4-4f48-a086-c62bc4e1d379","Type":"ContainerDied","Data":"b7d74161773ef6e8b2128614d9a9dab513f0cfb55bb4c058d21ff60e4ae447c9"} Oct 29 23:42:59.854304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.853895 5736 scope.go:122] "RemoveContainer" containerID="40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18" Oct 29 23:42:59.854304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.854055 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv" Oct 29 23:42:59.880491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.880461 5736 scope.go:122] "RemoveContainer" containerID="40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18" Oct 29 23:42:59.881183 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:42:59.881148 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18\": container with ID starting with 40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18 not found: ID does not exist" containerID="40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18" Oct 29 23:42:59.881358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.881332 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18"} err="failed to get container status \"40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18\": rpc error: code = NotFound desc = could not find container \"40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18\": container with ID starting with 40fc8bb8eb43434150cdbcc54408ef9039a436c5044ab9eae2f5410e6a99dd18 not found: ID does not exist" Oct 29 23:42:59.883975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.883955 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz"] Oct 29 23:42:59.886871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.886846 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d8dbbcec-24d4-4f48-a086-c62bc4e1d379" containerName="maas-api" Oct 29 23:42:59.887532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.887508 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8dbbcec-24d4-4f48-a086-c62bc4e1d379" containerName="maas-api" Oct 29 23:42:59.887739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.887723 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" Oct 29 23:42:59.887852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.887839 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecabf71e-aeb5-4a40-a591-1dfbb4f0468e" containerName="istio-proxy" Oct 29 23:42:59.888235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.888203 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8dbbcec-24d4-4f48-a086-c62bc4e1d379" containerName="maas-api" Oct 29 23:42:59.888445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.888418 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-metrics-certs\") pod \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " Oct 29 23:42:59.889256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.889233 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-kube-api-access-zgwm4\" (UniqueName: \"kubernetes.io/projected/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-kube-api-access-zgwm4\") pod \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " Oct 29 23:42:59.889860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.889839 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls\") pod \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " Oct 29 23:42:59.890664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.890642 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-openshift-service-ca\") pod \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\" (UID: \"d8dbbcec-24d4-4f48-a086-c62bc4e1d379\") " Oct 29 23:42:59.891382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.891278 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-openshift-service-ca" pod "d8dbbcec-24d4-4f48-a086-c62bc4e1d379" (UID: "d8dbbcec-24d4-4f48-a086-c62bc4e1d379"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:42:59.891694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.891657 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-kube-api-access-zgwm4" pod "d8dbbcec-24d4-4f48-a086-c62bc4e1d379" (UID: "d8dbbcec-24d4-4f48-a086-c62bc4e1d379"). InnerVolumeSpecName "kube-api-access-zgwm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:42:59.892456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.892425 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:59.892556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.892461 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgwm4\" (UniqueName: \"kubernetes.io/projected/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-kube-api-access-zgwm4\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:42:59.896672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.896643 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls" pod "d8dbbcec-24d4-4f48-a086-c62bc4e1d379" (UID: "d8dbbcec-24d4-4f48-a086-c62bc4e1d379"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:42:59.897964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.897926 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-metrics-certs" pod "d8dbbcec-24d4-4f48-a086-c62bc4e1d379" (UID: "d8dbbcec-24d4-4f48-a086-c62bc4e1d379"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:42:59.899421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.899399 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls"] Oct 29 23:42:59.899521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.899438 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c5ef2f-xzpls"] Oct 29 23:42:59.899521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.899506 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:42:59.915173 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.915147 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d20345-serving-cert\"" Oct 29 23:42:59.918828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:42:59.918801 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz"] Oct 29 23:43:00.017653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.017533 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-maas-api-tls\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.018095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.018065 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-metrics-certs\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.018598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.018507 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.018820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.018799 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvk6\" (UniqueName: \"kubernetes.io/projected/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-kube-api-access-psvk6\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.019111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.019094 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:00.019232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.019220 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dbbcec-24d4-4f48-a086-c62bc4e1d379-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:00.120834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.120794 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.120834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.120838 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvk6\" (UniqueName: \"kubernetes.io/projected/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-kube-api-access-psvk6\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.121100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.120956 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-maas-api-tls\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.121235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.121211 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-metrics-certs\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.121629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.121574 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.123768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.123742 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-maas-api-tls\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.123866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.123746 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-metrics-certs\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.164357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.164323 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvk6\" (UniqueName: \"kubernetes.io/projected/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-kube-api-access-psvk6\") pod \"maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.223351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.223290 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv"] Oct 29 23:43:00.227746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.227721 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c5ef2f-559d487c7-64nmv"] Oct 29 23:43:00.276257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.276186 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:00.555775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.554669 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:43:00.613941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.613908 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz"] Oct 29 23:43:00.615740 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:43:00.615714 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b11d7a_4eac_42c3_ba4e_ab8aaf081bc0.slice/crio-b3c1b6c9cf02ecac09ac1468be1c052f1443385f96a2b3fb756e15e3b9ec8ec8 WatchSource:0}: Error finding container b3c1b6c9cf02ecac09ac1468be1c052f1443385f96a2b3fb756e15e3b9ec8ec8: Status 404 returned error can't find the container with id b3c1b6c9cf02ecac09ac1468be1c052f1443385f96a2b3fb756e15e3b9ec8ec8 Oct 29 23:43:00.760878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.760838 5736 patch_prober.go:29] interesting pod/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" start-of-body= Oct 29 23:43:00.761066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.760903 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" Oct 29 23:43:00.868748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:00.868648 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" event={"ID":"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0","Type":"ContainerStarted","Data":"b3c1b6c9cf02ecac09ac1468be1c052f1443385f96a2b3fb756e15e3b9ec8ec8"} Oct 29 23:43:01.009299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:01.009269 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7" path="/var/lib/kubelet/pods/c53b91f3-0411-4d0d-a9c4-9f1e5034dcd7/volumes" Oct 29 23:43:01.009834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:01.009816 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8dbbcec-24d4-4f48-a086-c62bc4e1d379" path="/var/lib/kubelet/pods/d8dbbcec-24d4-4f48-a086-c62bc4e1d379/volumes" Oct 29 23:43:01.761010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:01.760969 5736 patch_prober.go:29] interesting pod/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" start-of-body= Oct 29 23:43:01.761212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:01.761036 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.118:15021/healthz/ready\": dial tcp 10.134.1.118:15021: connect: connection refused" Oct 29 23:43:01.880454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:01.880413 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" event={"ID":"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0","Type":"ContainerStarted","Data":"7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061"} Oct 29 23:43:01.880955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:01.880475 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:01.925950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:01.925897 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" podStartSLOduration=38.538804865 podStartE2EDuration="38.925883468s" totalImagesPullingTime="387.078603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:01.923095008 +0000 UTC m=+3579.039014616" watchObservedRunningTime="2026-10-29 23:43:01.925883468 +0000 UTC m=+3579.041803074" Oct 29 23:43:02.766566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:02.766534 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:43:02.767695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:02.767668 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:43:07.670224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:07.670188 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4"] Oct 29 23:43:07.671180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:07.671113 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" containerID="cri-o://4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d" gracePeriod=30 Oct 29 23:43:08.226821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.224465 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:43:08.500929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.500853 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm"] Oct 29 23:43:08.505900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.505878 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.523095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.523072 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:43:08.523095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.523081 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:43:08.528484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.528462 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm"] Oct 29 23:43:08.535550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.535526 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.535834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.535814 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.535997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.535972 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slgx2\" (UniqueName: \"kubernetes.io/projected/2832935d-1204-4b0d-b898-30ac03a097d6-kube-api-access-slgx2\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.637360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.637328 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.637673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.637651 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.637753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.637735 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slgx2\" (UniqueName: \"kubernetes.io/projected/2832935d-1204-4b0d-b898-30ac03a097d6-kube-api-access-slgx2\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.638396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.638342 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.639787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.639758 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.660913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.660886 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slgx2\" (UniqueName: \"kubernetes.io/projected/2832935d-1204-4b0d-b898-30ac03a097d6-kube-api-access-slgx2\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:08.821932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:08.821846 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:09.138380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.138353 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm"] Oct 29 23:43:09.140296 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:43:09.140269 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2832935d_1204_4b0d_b898_30ac03a097d6.slice/crio-f5e20f529c464a340da11c01f295a4e3fd1b3f4b51b786e306309174e9d96412 WatchSource:0}: Error finding container f5e20f529c464a340da11c01f295a4e3fd1b3f4b51b786e306309174e9d96412: Status 404 returned error can't find the container with id f5e20f529c464a340da11c01f295a4e3fd1b3f4b51b786e306309174e9d96412 Oct 29 23:43:09.192906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.192876 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" event={"ID":"2832935d-1204-4b0d-b898-30ac03a097d6","Type":"ContainerStarted","Data":"f5e20f529c464a340da11c01f295a4e3fd1b3f4b51b786e306309174e9d96412"} Oct 29 23:43:09.603292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.600090 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz"] Oct 29 23:43:09.608507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.608475 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.610569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.610537 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz"] Oct 29 23:43:09.677608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.676989 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-328cb7e6-kprxz\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.682102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.681608 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4sw\" (UniqueName: \"kubernetes.io/projected/cce93aaa-0f13-4687-b455-aed1a4fce1ce-kube-api-access-rt4sw\") pod \"maas-api-revoke-sub-328cb7e6-kprxz\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.682102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.681771 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-328cb7e6-kprxz\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.784104 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.784061 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4sw\" (UniqueName: \"kubernetes.io/projected/cce93aaa-0f13-4687-b455-aed1a4fce1ce-kube-api-access-rt4sw\") pod \"maas-api-revoke-sub-328cb7e6-kprxz\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.784310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.784174 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-328cb7e6-kprxz\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.784482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.784463 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-328cb7e6-kprxz\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.785262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.785240 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-328cb7e6-kprxz\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.786843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.786824 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-328cb7e6-kprxz\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.808046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.807995 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4sw\" (UniqueName: \"kubernetes.io/projected/cce93aaa-0f13-4687-b455-aed1a4fce1ce-kube-api-access-rt4sw\") pod \"maas-api-revoke-sub-328cb7e6-kprxz\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:09.948481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:09.948451 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:10.207368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:10.207278 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" event={"ID":"2832935d-1204-4b0d-b898-30ac03a097d6","Type":"ContainerStarted","Data":"651fa3be849b00b78155313178f783ef5c04d9c001ad340c85d8176120dc9d8b"} Oct 29 23:43:10.260254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:10.260227 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz"] Oct 29 23:43:10.262195 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:43:10.262164 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce93aaa_0f13_4687_b455_aed1a4fce1ce.slice/crio-47f5afe8262f77623affb87f7269b33d3e025fa039f6095d039e8685b0b501f8 WatchSource:0}: Error finding container 47f5afe8262f77623affb87f7269b33d3e025fa039f6095d039e8685b0b501f8: Status 404 returned error can't find the container with id 47f5afe8262f77623affb87f7269b33d3e025fa039f6095d039e8685b0b501f8 Oct 29 23:43:10.268919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:10.268850 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" podStartSLOduration=2.268837177 podStartE2EDuration="2.268837177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:10.254667181 +0000 UTC m=+3587.370586786" watchObservedRunningTime="2026-10-29 23:43:10.268837177 +0000 UTC m=+3587.384756782" Oct 29 23:43:11.219863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:11.219823 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" event={"ID":"cce93aaa-0f13-4687-b455-aed1a4fce1ce","Type":"ContainerStarted","Data":"b29ec2d62cf732738ba1ec0c378234026ae84fe7b6fdba31b80cfb30dacbca9d"} Oct 29 23:43:11.220349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:11.219892 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" event={"ID":"cce93aaa-0f13-4687-b455-aed1a4fce1ce","Type":"ContainerStarted","Data":"47f5afe8262f77623affb87f7269b33d3e025fa039f6095d039e8685b0b501f8"} Oct 29 23:43:11.221623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:11.221595 5736 generic.go:356] "Generic (PLEG): container finished" podID="2832935d-1204-4b0d-b898-30ac03a097d6" containerID="651fa3be849b00b78155313178f783ef5c04d9c001ad340c85d8176120dc9d8b" exitCode=0 Oct 29 23:43:11.221765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:11.221676 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" event={"ID":"2832935d-1204-4b0d-b898-30ac03a097d6","Type":"ContainerDied","Data":"651fa3be849b00b78155313178f783ef5c04d9c001ad340c85d8176120dc9d8b"} Oct 29 23:43:11.263449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:11.263402 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" podStartSLOduration=2.263389261 podStartE2EDuration="2.263389261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:11.259464997 +0000 UTC m=+3588.375384603" watchObservedRunningTime="2026-10-29 23:43:11.263389261 +0000 UTC m=+3588.379308867" Oct 29 23:43:11.358501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:11.358473 5736 patch_prober.go:29] interesting pod/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:43:11.358635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:11.358526 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:43:12.232799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.232760 5736 generic.go:356] "Generic (PLEG): container finished" podID="cce93aaa-0f13-4687-b455-aed1a4fce1ce" containerID="b29ec2d62cf732738ba1ec0c378234026ae84fe7b6fdba31b80cfb30dacbca9d" exitCode=0 Oct 29 23:43:12.233336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.232820 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" event={"ID":"cce93aaa-0f13-4687-b455-aed1a4fce1ce","Type":"ContainerDied","Data":"b29ec2d62cf732738ba1ec0c378234026ae84fe7b6fdba31b80cfb30dacbca9d"} Oct 29 23:43:12.855316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.855291 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:12.964497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.964464 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2832935d-1204-4b0d-b898-30ac03a097d6-kube-api-access-slgx2\" (UniqueName: \"kubernetes.io/projected/2832935d-1204-4b0d-b898-30ac03a097d6-kube-api-access-slgx2\") pod \"2832935d-1204-4b0d-b898-30ac03a097d6\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " Oct 29 23:43:12.964817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.964795 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-openshift-ca\") pod \"2832935d-1204-4b0d-b898-30ac03a097d6\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " Oct 29 23:43:12.964945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.964910 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-xks-ca\") pod \"2832935d-1204-4b0d-b898-30ac03a097d6\" (UID: \"2832935d-1204-4b0d-b898-30ac03a097d6\") " Oct 29 23:43:12.965236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.965205 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-openshift-ca" pod "2832935d-1204-4b0d-b898-30ac03a097d6" (UID: "2832935d-1204-4b0d-b898-30ac03a097d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:12.965731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.965709 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:12.966812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.966748 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2832935d-1204-4b0d-b898-30ac03a097d6-kube-api-access-slgx2" pod "2832935d-1204-4b0d-b898-30ac03a097d6" (UID: "2832935d-1204-4b0d-b898-30ac03a097d6"). InnerVolumeSpecName "kube-api-access-slgx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:12.966906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:12.966891 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-xks-ca" pod "2832935d-1204-4b0d-b898-30ac03a097d6" (UID: "2832935d-1204-4b0d-b898-30ac03a097d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:13.067002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.066964 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slgx2\" (UniqueName: \"kubernetes.io/projected/2832935d-1204-4b0d-b898-30ac03a097d6-kube-api-access-slgx2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.067002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.066997 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2832935d-1204-4b0d-b898-30ac03a097d6-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.243049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.242971 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" Oct 29 23:43:13.243049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.243006 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm" event={"ID":"2832935d-1204-4b0d-b898-30ac03a097d6","Type":"ContainerDied","Data":"f5e20f529c464a340da11c01f295a4e3fd1b3f4b51b786e306309174e9d96412"} Oct 29 23:43:13.243539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.243061 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5e20f529c464a340da11c01f295a4e3fd1b3f4b51b786e306309174e9d96412" Oct 29 23:43:13.404180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.404138 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:13.515950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.515922 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:43:13.575637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.575601 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-xks-ca\") pod \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " Oct 29 23:43:13.575833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.575811 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce93aaa-0f13-4687-b455-aed1a4fce1ce-kube-api-access-rt4sw\" (UniqueName: \"kubernetes.io/projected/cce93aaa-0f13-4687-b455-aed1a4fce1ce-kube-api-access-rt4sw\") pod \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " Oct 29 23:43:13.576050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.576035 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-openshift-ca\") pod \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\" (UID: \"cce93aaa-0f13-4687-b455-aed1a4fce1ce\") " Oct 29 23:43:13.576414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.576391 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-openshift-ca" pod "cce93aaa-0f13-4687-b455-aed1a4fce1ce" (UID: "cce93aaa-0f13-4687-b455-aed1a4fce1ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:13.576885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.576867 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.578328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.578304 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-xks-ca" pod "cce93aaa-0f13-4687-b455-aed1a4fce1ce" (UID: "cce93aaa-0f13-4687-b455-aed1a4fce1ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:13.578504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.578483 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce93aaa-0f13-4687-b455-aed1a4fce1ce-kube-api-access-rt4sw" pod "cce93aaa-0f13-4687-b455-aed1a4fce1ce" (UID: "cce93aaa-0f13-4687-b455-aed1a4fce1ce"). InnerVolumeSpecName "kube-api-access-rt4sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.678421 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-socket\") pod \"b04a1403-813a-492d-8472-978da4638bad\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.678474 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-data\") pod \"b04a1403-813a-492d-8472-978da4638bad\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.678555 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b04a1403-813a-492d-8472-978da4638bad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04a1403-813a-492d-8472-978da4638bad-istio-podinfo\") pod \"b04a1403-813a-492d-8472-978da4638bad\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.678629 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04a1403-813a-492d-8472-978da4638bad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04a1403-813a-492d-8472-978da4638bad-istiod-ca-cert\") pod \"b04a1403-813a-492d-8472-978da4638bad\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.678974 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-certs\") pod \"b04a1403-813a-492d-8472-978da4638bad\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.679027 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-credential-socket\") pod \"b04a1403-813a-492d-8472-978da4638bad\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.679164 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-envoy\") pod \"b04a1403-813a-492d-8472-978da4638bad\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.679240 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-istio-token\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-istio-token\") pod \"b04a1403-813a-492d-8472-978da4638bad\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.679394 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-kube-api-access-9fdcb\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-kube-api-access-9fdcb\") pod \"b04a1403-813a-492d-8472-978da4638bad\" (UID: \"b04a1403-813a-492d-8472-978da4638bad\") " Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.679742 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04a1403-813a-492d-8472-978da4638bad-istiod-ca-cert" pod "b04a1403-813a-492d-8472-978da4638bad" (UID: "b04a1403-813a-492d-8472-978da4638bad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.680792 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce93aaa-0f13-4687-b455-aed1a4fce1ce-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.680816 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04a1403-813a-492d-8472-978da4638bad-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.680831 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4sw\" (UniqueName: \"kubernetes.io/projected/cce93aaa-0f13-4687-b455-aed1a4fce1ce-kube-api-access-rt4sw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.681013 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-socket" pod "b04a1403-813a-492d-8472-978da4638bad" (UID: "b04a1403-813a-492d-8472-978da4638bad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.681231 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-data" pod "b04a1403-813a-492d-8472-978da4638bad" (UID: "b04a1403-813a-492d-8472-978da4638bad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:13.683668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.683510 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-certs" pod "b04a1403-813a-492d-8472-978da4638bad" (UID: "b04a1403-813a-492d-8472-978da4638bad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:13.684756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.683729 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-credential-socket" pod "b04a1403-813a-492d-8472-978da4638bad" (UID: "b04a1403-813a-492d-8472-978da4638bad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:13.686379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.685794 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-envoy" pod "b04a1403-813a-492d-8472-978da4638bad" (UID: "b04a1403-813a-492d-8472-978da4638bad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:13.691625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.688298 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-istio-token" pod "b04a1403-813a-492d-8472-978da4638bad" (UID: "b04a1403-813a-492d-8472-978da4638bad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:13.691625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.688528 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-kube-api-access-9fdcb" pod "b04a1403-813a-492d-8472-978da4638bad" (UID: "b04a1403-813a-492d-8472-978da4638bad"). InnerVolumeSpecName "kube-api-access-9fdcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:13.695794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.695760 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b04a1403-813a-492d-8472-978da4638bad-istio-podinfo" pod "b04a1403-813a-492d-8472-978da4638bad" (UID: "b04a1403-813a-492d-8472-978da4638bad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:43:13.782960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.782885 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04a1403-813a-492d-8472-978da4638bad-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.782960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.782913 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.782960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.782923 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.782960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.782933 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.782960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.782942 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.782960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.782950 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fdcb\" (UniqueName: \"kubernetes.io/projected/b04a1403-813a-492d-8472-978da4638bad-kube-api-access-9fdcb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.782960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.782962 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:13.783309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:13.782972 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04a1403-813a-492d-8472-978da4638bad-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:14.252867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.252832 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" Oct 29 23:43:14.252867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.252853 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz" event={"ID":"cce93aaa-0f13-4687-b455-aed1a4fce1ce","Type":"ContainerDied","Data":"47f5afe8262f77623affb87f7269b33d3e025fa039f6095d039e8685b0b501f8"} Oct 29 23:43:14.253358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.252913 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47f5afe8262f77623affb87f7269b33d3e025fa039f6095d039e8685b0b501f8" Oct 29 23:43:14.254815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.254787 5736 generic.go:356] "Generic (PLEG): container finished" podID="b04a1403-813a-492d-8472-978da4638bad" containerID="4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d" exitCode=0 Oct 29 23:43:14.254929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.254857 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" Oct 29 23:43:14.254929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.254873 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" event={"ID":"b04a1403-813a-492d-8472-978da4638bad","Type":"ContainerDied","Data":"4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d"} Oct 29 23:43:14.254929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.254916 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4" event={"ID":"b04a1403-813a-492d-8472-978da4638bad","Type":"ContainerDied","Data":"570e86e0b4e02d4925e0be4773903a41190b2c5ee235681d3e259cb7119d7f42"} Oct 29 23:43:14.255081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.254940 5736 scope.go:122] "RemoveContainer" containerID="4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d" Oct 29 23:43:14.267776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.267758 5736 scope.go:122] "RemoveContainer" containerID="a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01" Oct 29 23:43:14.278557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.278536 5736 scope.go:122] "RemoveContainer" containerID="4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d" Oct 29 23:43:14.278909 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:43:14.278880 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d\": container with ID starting with 4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d not found: ID does not exist" containerID="4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d" Oct 29 23:43:14.279007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.278911 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d"} err="failed to get container status \"4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d\": rpc error: code = NotFound desc = could not find container \"4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d\": container with ID starting with 4bb33c7eb6af0127f258278a1ef9d5c9ffcc5b574aa53380e239fdca210afe6d not found: ID does not exist" Oct 29 23:43:14.279007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.278929 5736 scope.go:122] "RemoveContainer" containerID="a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01" Oct 29 23:43:14.279205 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:43:14.279186 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01\": container with ID starting with a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01 not found: ID does not exist" containerID="a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01" Oct 29 23:43:14.279251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.279216 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01"} err="failed to get container status \"a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01\": rpc error: code = NotFound desc = could not find container \"a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01\": container with ID starting with a60f1b00641311942c1c04502e180a0770065fa48651b4bd6467d728be4e7d01 not found: ID does not exist" Oct 29 23:43:14.314028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.314005 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4"] Oct 29 23:43:14.323255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:14.323233 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c5ef2f-openshift-default-7f8b87fd9-xqmf4"] Oct 29 23:43:15.005935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:15.005903 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04a1403-813a-492d-8472-978da4638bad" path="/var/lib/kubelet/pods/b04a1403-813a-492d-8472-978da4638bad/volumes" Oct 29 23:43:18.951085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:18.951052 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l"] Oct 29 23:43:20.067123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.067083 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45"] Oct 29 23:43:20.067920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.067902 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2832935d-1204-4b0d-b898-30ac03a097d6" containerName="revoke-keys" Oct 29 23:43:20.067920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.067919 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2832935d-1204-4b0d-b898-30ac03a097d6" containerName="revoke-keys" Oct 29 23:43:20.068043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.067964 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" Oct 29 23:43:20.068043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.067970 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" Oct 29 23:43:20.068043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.067987 5736 cpu_manager.go:414] "containerMap: removing container" podUID="cce93aaa-0f13-4687-b455-aed1a4fce1ce" containerName="revoke-keys" Oct 29 23:43:20.068043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.067992 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce93aaa-0f13-4687-b455-aed1a4fce1ce" containerName="revoke-keys" Oct 29 23:43:20.068043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.068001 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" Oct 29 23:43:20.068043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.068005 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" Oct 29 23:43:20.068247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.068113 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" Oct 29 23:43:20.068247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.068125 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2832935d-1204-4b0d-b898-30ac03a097d6" containerName="revoke-keys" Oct 29 23:43:20.068247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.068141 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce93aaa-0f13-4687-b455-aed1a4fce1ce" containerName="revoke-keys" Oct 29 23:43:20.072958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.072939 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.089064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.089039 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:43:20.094485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.094459 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45"] Oct 29 23:43:20.099571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.099548 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:43:20.133265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.133223 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.133451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.133406 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.134128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.134097 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxmq9\" (UniqueName: \"kubernetes.io/projected/66753b21-2e82-40e9-a9fe-ae6cc31995a9-kube-api-access-nxmq9\") pod \"maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.236063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.236027 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.236264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.236244 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxmq9\" (UniqueName: \"kubernetes.io/projected/66753b21-2e82-40e9-a9fe-ae6cc31995a9-kube-api-access-nxmq9\") pod \"maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.236483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.236463 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.237096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.237076 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.238423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.238397 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.259412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.259387 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxmq9\" (UniqueName: \"kubernetes.io/projected/66753b21-2e82-40e9-a9fe-ae6cc31995a9-kube-api-access-nxmq9\") pod \"maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.387829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.387756 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:20.708170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:20.708140 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45"] Oct 29 23:43:20.711185 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:43:20.711154 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66753b21_2e82_40e9_a9fe_ae6cc31995a9.slice/crio-37ddeeabb168e568db6c96cce7f515df7d6b6debd0288d15b91dc4bf9cbb03fa WatchSource:0}: Error finding container 37ddeeabb168e568db6c96cce7f515df7d6b6debd0288d15b91dc4bf9cbb03fa: Status 404 returned error can't find the container with id 37ddeeabb168e568db6c96cce7f515df7d6b6debd0288d15b91dc4bf9cbb03fa Oct 29 23:43:21.326157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.326114 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" event={"ID":"66753b21-2e82-40e9-a9fe-ae6cc31995a9","Type":"ContainerStarted","Data":"9076fa0cc96a70f144079fcd1aeb0258379f5813c94f4a41c1044e3bbabb7484"} Oct 29 23:43:21.326157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.326176 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" event={"ID":"66753b21-2e82-40e9-a9fe-ae6cc31995a9","Type":"ContainerStarted","Data":"37ddeeabb168e568db6c96cce7f515df7d6b6debd0288d15b91dc4bf9cbb03fa"} Oct 29 23:43:21.371508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.371407 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" podStartSLOduration=1.371392113 podStartE2EDuration="1.371392113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:21.36709848 +0000 UTC m=+3598.483018097" watchObservedRunningTime="2026-10-29 23:43:21.371392113 +0000 UTC m=+3598.487311713" Oct 29 23:43:21.517154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.517107 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79"] Oct 29 23:43:21.520378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.520344 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04a1403-813a-492d-8472-978da4638bad" containerName="istio-proxy" Oct 29 23:43:21.526340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.526309 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.542078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.542025 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79"] Oct 29 23:43:21.572146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.572113 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-899fc1a9-xzm79\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.572300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.572285 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjdl\" (UniqueName: \"kubernetes.io/projected/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-kube-api-access-jvjdl\") pod \"maas-api-revoke-sub-899fc1a9-xzm79\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.572471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.572442 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-899fc1a9-xzm79\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.674257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.674174 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-899fc1a9-xzm79\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.674420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.674295 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjdl\" (UniqueName: \"kubernetes.io/projected/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-kube-api-access-jvjdl\") pod \"maas-api-revoke-sub-899fc1a9-xzm79\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.674497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.674416 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-899fc1a9-xzm79\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.675297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.675268 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-899fc1a9-xzm79\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.676743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.676716 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-899fc1a9-xzm79\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.697372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.697345 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjdl\" (UniqueName: \"kubernetes.io/projected/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-kube-api-access-jvjdl\") pod \"maas-api-revoke-sub-899fc1a9-xzm79\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:21.848734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:21.848705 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:22.175080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:22.175051 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79"] Oct 29 23:43:22.176979 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:43:22.176943 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e7ee5f_b2de_4f4a_8a3f_57d761c57c8f.slice/crio-9d2ea1179d824b015650f2a6ddd07027fba9b9334a4705b8e9a9d2b922a68104 WatchSource:0}: Error finding container 9d2ea1179d824b015650f2a6ddd07027fba9b9334a4705b8e9a9d2b922a68104: Status 404 returned error can't find the container with id 9d2ea1179d824b015650f2a6ddd07027fba9b9334a4705b8e9a9d2b922a68104 Oct 29 23:43:22.339972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:22.339941 5736 generic.go:356] "Generic (PLEG): container finished" podID="66753b21-2e82-40e9-a9fe-ae6cc31995a9" containerID="9076fa0cc96a70f144079fcd1aeb0258379f5813c94f4a41c1044e3bbabb7484" exitCode=0 Oct 29 23:43:22.340412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:22.340021 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" event={"ID":"66753b21-2e82-40e9-a9fe-ae6cc31995a9","Type":"ContainerDied","Data":"9076fa0cc96a70f144079fcd1aeb0258379f5813c94f4a41c1044e3bbabb7484"} Oct 29 23:43:22.341691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:22.341662 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" event={"ID":"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f","Type":"ContainerStarted","Data":"9d2ea1179d824b015650f2a6ddd07027fba9b9334a4705b8e9a9d2b922a68104"} Oct 29 23:43:23.353129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.353086 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" event={"ID":"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f","Type":"ContainerStarted","Data":"92407076100e03b3c11600243d977129dd260723afb148af2d5a29eff577c61a"} Oct 29 23:43:23.398194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.398138 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" podStartSLOduration=2.398119056 podStartE2EDuration="2.398119056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:23.392715122 +0000 UTC m=+3600.508634730" watchObservedRunningTime="2026-10-29 23:43:23.398119056 +0000 UTC m=+3600.514038665" Oct 29 23:43:23.541965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.541936 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:23.617545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.617475 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-xks-ca\") pod \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " Oct 29 23:43:23.617700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.617635 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66753b21-2e82-40e9-a9fe-ae6cc31995a9-kube-api-access-nxmq9\" (UniqueName: \"kubernetes.io/projected/66753b21-2e82-40e9-a9fe-ae6cc31995a9-kube-api-access-nxmq9\") pod \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " Oct 29 23:43:23.617744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.617693 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-openshift-ca\") pod \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\" (UID: \"66753b21-2e82-40e9-a9fe-ae6cc31995a9\") " Oct 29 23:43:23.618142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.618113 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-openshift-ca" pod "66753b21-2e82-40e9-a9fe-ae6cc31995a9" (UID: "66753b21-2e82-40e9-a9fe-ae6cc31995a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:23.618634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.618608 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:23.619818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.619786 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66753b21-2e82-40e9-a9fe-ae6cc31995a9-kube-api-access-nxmq9" pod "66753b21-2e82-40e9-a9fe-ae6cc31995a9" (UID: "66753b21-2e82-40e9-a9fe-ae6cc31995a9"). InnerVolumeSpecName "kube-api-access-nxmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:23.620092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.620063 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-xks-ca" pod "66753b21-2e82-40e9-a9fe-ae6cc31995a9" (UID: "66753b21-2e82-40e9-a9fe-ae6cc31995a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:23.720624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.720590 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxmq9\" (UniqueName: \"kubernetes.io/projected/66753b21-2e82-40e9-a9fe-ae6cc31995a9-kube-api-access-nxmq9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:23.720624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.720620 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66753b21-2e82-40e9-a9fe-ae6cc31995a9-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:23.951471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:23.951385 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" podUID="604aee36-959a-4dd8-8b9c-108e22cf2649" containerName="payload-processing" containerID="cri-o://96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7" gracePeriod=25 Oct 29 23:43:24.102854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.102820 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:43:24.141175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.141020 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:43:24.223314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.223284 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:43:24.329638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.329601 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604aee36-959a-4dd8-8b9c-108e22cf2649-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604aee36-959a-4dd8-8b9c-108e22cf2649-plugins-config-volume\") pod \"604aee36-959a-4dd8-8b9c-108e22cf2649\" (UID: \"604aee36-959a-4dd8-8b9c-108e22cf2649\") " Oct 29 23:43:24.329832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.329795 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604aee36-959a-4dd8-8b9c-108e22cf2649-kube-api-access-lqglx\" (UniqueName: \"kubernetes.io/projected/604aee36-959a-4dd8-8b9c-108e22cf2649-kube-api-access-lqglx\") pod \"604aee36-959a-4dd8-8b9c-108e22cf2649\" (UID: \"604aee36-959a-4dd8-8b9c-108e22cf2649\") " Oct 29 23:43:24.330294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.330243 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604aee36-959a-4dd8-8b9c-108e22cf2649-plugins-config-volume" pod "604aee36-959a-4dd8-8b9c-108e22cf2649" (UID: "604aee36-959a-4dd8-8b9c-108e22cf2649"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:24.330730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.330707 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604aee36-959a-4dd8-8b9c-108e22cf2649-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:24.332880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.332854 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604aee36-959a-4dd8-8b9c-108e22cf2649-kube-api-access-lqglx" pod "604aee36-959a-4dd8-8b9c-108e22cf2649" (UID: "604aee36-959a-4dd8-8b9c-108e22cf2649"). InnerVolumeSpecName "kube-api-access-lqglx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:24.368454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.368416 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" Oct 29 23:43:24.368946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.368420 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45" event={"ID":"66753b21-2e82-40e9-a9fe-ae6cc31995a9","Type":"ContainerDied","Data":"37ddeeabb168e568db6c96cce7f515df7d6b6debd0288d15b91dc4bf9cbb03fa"} Oct 29 23:43:24.369058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.369003 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37ddeeabb168e568db6c96cce7f515df7d6b6debd0288d15b91dc4bf9cbb03fa" Oct 29 23:43:24.370541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.370511 5736 generic.go:356] "Generic (PLEG): container finished" podID="a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f" containerID="92407076100e03b3c11600243d977129dd260723afb148af2d5a29eff577c61a" exitCode=0 Oct 29 23:43:24.370689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.370661 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" event={"ID":"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f","Type":"ContainerDied","Data":"92407076100e03b3c11600243d977129dd260723afb148af2d5a29eff577c61a"} Oct 29 23:43:24.374127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.374101 5736 generic.go:356] "Generic (PLEG): container finished" podID="604aee36-959a-4dd8-8b9c-108e22cf2649" containerID="96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7" exitCode=0 Oct 29 23:43:24.374241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.374139 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" Oct 29 23:43:24.374529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.374326 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" event={"ID":"604aee36-959a-4dd8-8b9c-108e22cf2649","Type":"ContainerDied","Data":"96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7"} Oct 29 23:43:24.374529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.374371 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l" event={"ID":"604aee36-959a-4dd8-8b9c-108e22cf2649","Type":"ContainerDied","Data":"f19e81d1cac018a4693af7bd775a58ec24d0d21ce17eb47f2237b673a6219f09"} Oct 29 23:43:24.374529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.374404 5736 scope.go:122] "RemoveContainer" containerID="96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7" Oct 29 23:43:24.408955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.400717 5736 scope.go:122] "RemoveContainer" containerID="96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7" Oct 29 23:43:24.408955 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:43:24.402924 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7\": container with ID starting with 96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7 not found: ID does not exist" containerID="96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7" Oct 29 23:43:24.408955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.402964 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7"} err="failed to get container status \"96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7\": rpc error: code = NotFound desc = could not find container \"96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7\": container with ID starting with 96d9bef21b23d3ae29088e5e5fde013d94a3f2879339ba25e4508774943adcb7 not found: ID does not exist" Oct 29 23:43:24.432443 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.432416 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqglx\" (UniqueName: \"kubernetes.io/projected/604aee36-959a-4dd8-8b9c-108e22cf2649-kube-api-access-lqglx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:24.451456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.451426 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l"] Oct 29 23:43:24.456760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:24.456731 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e482-67b68f95cd-bnh6l"] Oct 29 23:43:25.006654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.006616 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604aee36-959a-4dd8-8b9c-108e22cf2649" path="/var/lib/kubelet/pods/604aee36-959a-4dd8-8b9c-108e22cf2649/volumes" Oct 29 23:43:25.564523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.564495 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:25.672369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.672289 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-openshift-ca\") pod \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " Oct 29 23:43:25.672535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.672424 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-xks-ca\") pod \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " Oct 29 23:43:25.672702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.672538 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-kube-api-access-jvjdl\" (UniqueName: \"kubernetes.io/projected/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-kube-api-access-jvjdl\") pod \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\" (UID: \"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f\") " Oct 29 23:43:25.672780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.672695 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-openshift-ca" pod "a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f" (UID: "a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:25.673533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.673510 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:25.674721 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.674698 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-xks-ca" pod "a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f" (UID: "a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:25.674893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.674872 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-kube-api-access-jvjdl" pod "a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f" (UID: "a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f"). InnerVolumeSpecName "kube-api-access-jvjdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:25.775747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.775716 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:25.775747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:25.775750 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvjdl\" (UniqueName: \"kubernetes.io/projected/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f-kube-api-access-jvjdl\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:26.396935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:26.396904 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" Oct 29 23:43:26.397145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:26.396886 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79" event={"ID":"a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f","Type":"ContainerDied","Data":"9d2ea1179d824b015650f2a6ddd07027fba9b9334a4705b8e9a9d2b922a68104"} Oct 29 23:43:26.397145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:26.397035 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d2ea1179d824b015650f2a6ddd07027fba9b9334a4705b8e9a9d2b922a68104" Oct 29 23:43:26.825309 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:43:26.825272 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:43:29.224638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:29.224605 5736 scope.go:122] "RemoveContainer" containerID="b73d9e554077fc3fe54169e1db0e3851dabb7664f45ca5d5c6753958772ae007" Oct 29 23:43:29.236179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:29.236157 5736 scope.go:122] "RemoveContainer" containerID="12d3dc22b53907cf8e7e89978243599445203854fac3a0e620ffaaaca4908b49" Oct 29 23:43:29.246777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:29.246753 5736 scope.go:122] "RemoveContainer" containerID="1eb65f822692f092b137800f04aa10e669327cac99cba7f9ed28014fcf121550" Oct 29 23:43:29.256775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:29.256754 5736 scope.go:122] "RemoveContainer" containerID="f8cb3749c0f3ad829dffa9348470a479a96777e4667c2738c4ba065a460bcc5c" Oct 29 23:43:29.266729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:29.266708 5736 scope.go:122] "RemoveContainer" containerID="31b9e3021ca113109584a274ee718e97850439dc7bbf04d645bce7d515a4fbe4" Oct 29 23:43:29.276519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:29.276496 5736 scope.go:122] "RemoveContainer" containerID="6b74cae0e513ec2b3b60cfe091f6cc44d52d694b95bbdbcf976c90d9994b6ace" Oct 29 23:43:29.286270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:29.286250 5736 scope.go:122] "RemoveContainer" containerID="fc988303a88d1938804372ae51cdd6f43de597a646100e9bdc696a3f608af5ff" Oct 29 23:43:36.031399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:36.031308 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6"] Oct 29 23:43:36.042789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:36.042751 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc10bf0-rkhc6"] Oct 29 23:43:37.006640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:37.006601 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa40060-1336-4b82-96aa-e1cb620c4ec5" path="/var/lib/kubelet/pods/6fa40060-1336-4b82-96aa-e1cb620c4ec5/volumes" Oct 29 23:43:46.764718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:46.764675 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr"] Oct 29 23:43:47.069311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.069215 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q"] Oct 29 23:43:47.089158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.086826 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ad63b16-9tg2q"] Oct 29 23:43:47.113127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.113092 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs"] Oct 29 23:43:47.113982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.113963 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f" containerName="revoke-keys" Oct 29 23:43:47.114058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.113982 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f" containerName="revoke-keys" Oct 29 23:43:47.114058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.113997 5736 cpu_manager.go:414] "containerMap: removing container" podUID="66753b21-2e82-40e9-a9fe-ae6cc31995a9" containerName="revoke-keys" Oct 29 23:43:47.114058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.114004 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66753b21-2e82-40e9-a9fe-ae6cc31995a9" containerName="revoke-keys" Oct 29 23:43:47.114058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.114020 5736 cpu_manager.go:414] "containerMap: removing container" podUID="604aee36-959a-4dd8-8b9c-108e22cf2649" containerName="payload-processing" Oct 29 23:43:47.114058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.114025 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604aee36-959a-4dd8-8b9c-108e22cf2649" containerName="payload-processing" Oct 29 23:43:47.114234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.114204 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f" containerName="revoke-keys" Oct 29 23:43:47.114234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.114214 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="66753b21-2e82-40e9-a9fe-ae6cc31995a9" containerName="revoke-keys" Oct 29 23:43:47.114234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.114223 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="604aee36-959a-4dd8-8b9c-108e22cf2649" containerName="payload-processing" Oct 29 23:43:47.120123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.120103 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.135565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.135541 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:43:47.135767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.135683 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:43:47.140086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.140060 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs"] Oct 29 23:43:47.170918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.170879 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.171143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.170941 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx7qf\" (UniqueName: \"kubernetes.io/projected/4b8e9925-6fef-4111-be0c-b6bd3147830d-kube-api-access-sx7qf\") pod \"maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.171292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.171259 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.273433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.273388 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.273672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.273447 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx7qf\" (UniqueName: \"kubernetes.io/projected/4b8e9925-6fef-4111-be0c-b6bd3147830d-kube-api-access-sx7qf\") pod \"maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.273751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.273713 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.274207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.274180 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.276209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.276186 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.297679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.297656 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx7qf\" (UniqueName: \"kubernetes.io/projected/4b8e9925-6fef-4111-be0c-b6bd3147830d-kube-api-access-sx7qf\") pod \"maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.436093 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.436022 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:47.756268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:47.756232 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs"] Oct 29 23:43:47.758279 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:43:47.758246 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8e9925_6fef_4111_be0c_b6bd3147830d.slice/crio-dea038543145b252e5075eb7f0d34c3ab7ce5617724cae74ec9b8ffedc4538d4 WatchSource:0}: Error finding container dea038543145b252e5075eb7f0d34c3ab7ce5617724cae74ec9b8ffedc4538d4: Status 404 returned error can't find the container with id dea038543145b252e5075eb7f0d34c3ab7ce5617724cae74ec9b8ffedc4538d4 Oct 29 23:43:48.739854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:48.739813 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" event={"ID":"4b8e9925-6fef-4111-be0c-b6bd3147830d","Type":"ContainerStarted","Data":"a0122019a4cbac13ce9f1cbe60b34b4d2931aaebbae6d27c74ac9920c4c25395"} Oct 29 23:43:48.740308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:48.739911 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" event={"ID":"4b8e9925-6fef-4111-be0c-b6bd3147830d","Type":"ContainerStarted","Data":"dea038543145b252e5075eb7f0d34c3ab7ce5617724cae74ec9b8ffedc4538d4"} Oct 29 23:43:48.781635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:48.781567 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" podStartSLOduration=1.781554026 podStartE2EDuration="1.781554026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:48.779313045 +0000 UTC m=+3625.895232651" watchObservedRunningTime="2026-10-29 23:43:48.781554026 +0000 UTC m=+3625.897473656" Oct 29 23:43:49.006690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:49.006620 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38a56c6-a367-4443-b2d7-c8777d8a4e21" path="/var/lib/kubelet/pods/a38a56c6-a367-4443-b2d7-c8777d8a4e21/volumes" Oct 29 23:43:49.815602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:49.814686 5736 generic.go:356] "Generic (PLEG): container finished" podID="4b8e9925-6fef-4111-be0c-b6bd3147830d" containerID="a0122019a4cbac13ce9f1cbe60b34b4d2931aaebbae6d27c74ac9920c4c25395" exitCode=0 Oct 29 23:43:49.815602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:49.815066 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" event={"ID":"4b8e9925-6fef-4111-be0c-b6bd3147830d","Type":"ContainerDied","Data":"a0122019a4cbac13ce9f1cbe60b34b4d2931aaebbae6d27c74ac9920c4c25395"} Oct 29 23:43:51.317662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.317630 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:51.454534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.454502 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-xks-ca\") pod \"4b8e9925-6fef-4111-be0c-b6bd3147830d\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " Oct 29 23:43:51.454781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.454558 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-openshift-ca\") pod \"4b8e9925-6fef-4111-be0c-b6bd3147830d\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " Oct 29 23:43:51.454781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.454678 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8e9925-6fef-4111-be0c-b6bd3147830d-kube-api-access-sx7qf\" (UniqueName: \"kubernetes.io/projected/4b8e9925-6fef-4111-be0c-b6bd3147830d-kube-api-access-sx7qf\") pod \"4b8e9925-6fef-4111-be0c-b6bd3147830d\" (UID: \"4b8e9925-6fef-4111-be0c-b6bd3147830d\") " Oct 29 23:43:51.455006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.454978 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-openshift-ca" pod "4b8e9925-6fef-4111-be0c-b6bd3147830d" (UID: "4b8e9925-6fef-4111-be0c-b6bd3147830d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:51.455740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.455711 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:51.457442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.457302 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-xks-ca" pod "4b8e9925-6fef-4111-be0c-b6bd3147830d" (UID: "4b8e9925-6fef-4111-be0c-b6bd3147830d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:51.457442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.457375 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8e9925-6fef-4111-be0c-b6bd3147830d-kube-api-access-sx7qf" pod "4b8e9925-6fef-4111-be0c-b6bd3147830d" (UID: "4b8e9925-6fef-4111-be0c-b6bd3147830d"). InnerVolumeSpecName "kube-api-access-sx7qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:51.557842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.557755 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8e9925-6fef-4111-be0c-b6bd3147830d-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:51.557842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.557794 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx7qf\" (UniqueName: \"kubernetes.io/projected/4b8e9925-6fef-4111-be0c-b6bd3147830d-kube-api-access-sx7qf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:43:51.889659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.879541 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" event={"ID":"4b8e9925-6fef-4111-be0c-b6bd3147830d","Type":"ContainerDied","Data":"dea038543145b252e5075eb7f0d34c3ab7ce5617724cae74ec9b8ffedc4538d4"} Oct 29 23:43:51.889659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.879619 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea038543145b252e5075eb7f0d34c3ab7ce5617724cae74ec9b8ffedc4538d4" Oct 29 23:43:51.889659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:51.879687 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs" Oct 29 23:43:56.828702 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:43:56.828663 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:43:57.435090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:43:57.435053 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz"] Oct 29 23:44:01.800976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:01.800932 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" podUID="55389672-b516-4422-b514-3ca1355c361a" containerName="main" containerID="cri-o://7068750864e51e7939aa7b83ce60996ed0f9ce402437b4a0b9b2e271a07bd9e3" gracePeriod=45 Oct 29 23:44:01.975984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:01.975949 5736 generic.go:356] "Generic (PLEG): container finished" podID="55389672-b516-4422-b514-3ca1355c361a" containerID="7068750864e51e7939aa7b83ce60996ed0f9ce402437b4a0b9b2e271a07bd9e3" exitCode=0 Oct 29 23:44:01.976152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:01.976018 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" event={"ID":"55389672-b516-4422-b514-3ca1355c361a","Type":"ContainerDied","Data":"7068750864e51e7939aa7b83ce60996ed0f9ce402437b4a0b9b2e271a07bd9e3"} Oct 29 23:44:02.110624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.110555 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:44:02.161234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.161202 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55389672-b516-4422-b514-3ca1355c361a-kube-api-access-t2nrf\" (UniqueName: \"kubernetes.io/projected/55389672-b516-4422-b514-3ca1355c361a-kube-api-access-t2nrf\") pod \"55389672-b516-4422-b514-3ca1355c361a\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " Oct 29 23:44:02.161431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.161266 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-tmp-dir\") pod \"55389672-b516-4422-b514-3ca1355c361a\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " Oct 29 23:44:02.161431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.161310 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-home\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-home\") pod \"55389672-b516-4422-b514-3ca1355c361a\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " Oct 29 23:44:02.161431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.161422 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-model-cache\") pod \"55389672-b516-4422-b514-3ca1355c361a\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " Oct 29 23:44:02.161639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.161512 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55389672-b516-4422-b514-3ca1355c361a-tls-certs\" (UniqueName: \"kubernetes.io/secret/55389672-b516-4422-b514-3ca1355c361a-tls-certs\") pod \"55389672-b516-4422-b514-3ca1355c361a\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " Oct 29 23:44:02.161639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.161551 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-dshm\") pod \"55389672-b516-4422-b514-3ca1355c361a\" (UID: \"55389672-b516-4422-b514-3ca1355c361a\") " Oct 29 23:44:02.161639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.161617 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-tmp-dir" pod "55389672-b516-4422-b514-3ca1355c361a" (UID: "55389672-b516-4422-b514-3ca1355c361a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:02.162323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.161666 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-home" pod "55389672-b516-4422-b514-3ca1355c361a" (UID: "55389672-b516-4422-b514-3ca1355c361a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:02.162323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.161740 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-model-cache" pod "55389672-b516-4422-b514-3ca1355c361a" (UID: "55389672-b516-4422-b514-3ca1355c361a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:02.162323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.162309 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:02.162514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.162326 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:02.162514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.162336 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:02.163861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.163829 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55389672-b516-4422-b514-3ca1355c361a-tls-certs" pod "55389672-b516-4422-b514-3ca1355c361a" (UID: "55389672-b516-4422-b514-3ca1355c361a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:02.164011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.163991 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-dshm" pod "55389672-b516-4422-b514-3ca1355c361a" (UID: "55389672-b516-4422-b514-3ca1355c361a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:02.164078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.164054 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55389672-b516-4422-b514-3ca1355c361a-kube-api-access-t2nrf" pod "55389672-b516-4422-b514-3ca1355c361a" (UID: "55389672-b516-4422-b514-3ca1355c361a"). InnerVolumeSpecName "kube-api-access-t2nrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:02.264520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.264485 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2nrf\" (UniqueName: \"kubernetes.io/projected/55389672-b516-4422-b514-3ca1355c361a-kube-api-access-t2nrf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:02.264520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.264518 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55389672-b516-4422-b514-3ca1355c361a-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:02.264520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.264529 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55389672-b516-4422-b514-3ca1355c361a-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:02.436334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.436257 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" podUID="75b35f2a-9440-48e6-b6a3-52dd58320ed0" containerName="payload-processing" containerID="cri-o://dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48" gracePeriod=25 Oct 29 23:44:02.714410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.714382 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:44:02.775182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.775148 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b35f2a-9440-48e6-b6a3-52dd58320ed0-kube-api-access-gz2jh\" (UniqueName: \"kubernetes.io/projected/75b35f2a-9440-48e6-b6a3-52dd58320ed0-kube-api-access-gz2jh\") pod \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\" (UID: \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\") " Oct 29 23:44:02.775630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.775563 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b35f2a-9440-48e6-b6a3-52dd58320ed0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b35f2a-9440-48e6-b6a3-52dd58320ed0-plugins-config-volume\") pod \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\" (UID: \"75b35f2a-9440-48e6-b6a3-52dd58320ed0\") " Oct 29 23:44:02.776753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.776724 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b35f2a-9440-48e6-b6a3-52dd58320ed0-plugins-config-volume" pod "75b35f2a-9440-48e6-b6a3-52dd58320ed0" (UID: "75b35f2a-9440-48e6-b6a3-52dd58320ed0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:02.778089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.778060 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b35f2a-9440-48e6-b6a3-52dd58320ed0-kube-api-access-gz2jh" pod "75b35f2a-9440-48e6-b6a3-52dd58320ed0" (UID: "75b35f2a-9440-48e6-b6a3-52dd58320ed0"). InnerVolumeSpecName "kube-api-access-gz2jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:02.877901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.877865 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz2jh\" (UniqueName: \"kubernetes.io/projected/75b35f2a-9440-48e6-b6a3-52dd58320ed0-kube-api-access-gz2jh\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:02.877901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:02.877898 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b35f2a-9440-48e6-b6a3-52dd58320ed0-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:03.066328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.066288 5736 generic.go:356] "Generic (PLEG): container finished" podID="75b35f2a-9440-48e6-b6a3-52dd58320ed0" containerID="dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48" exitCode=0 Oct 29 23:44:03.071473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.071439 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" Oct 29 23:44:03.072075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.072052 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" Oct 29 23:44:03.073956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.073910 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" event={"ID":"75b35f2a-9440-48e6-b6a3-52dd58320ed0","Type":"ContainerDied","Data":"dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48"} Oct 29 23:44:03.074088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.073972 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz" event={"ID":"75b35f2a-9440-48e6-b6a3-52dd58320ed0","Type":"ContainerDied","Data":"c72801f4139a18bff7d2e0c66118ae02a863413f587446cc930e63bba16aa940"} Oct 29 23:44:03.074088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.073996 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr" event={"ID":"55389672-b516-4422-b514-3ca1355c361a","Type":"ContainerDied","Data":"95445d33f2831cf9d76a1542bb577d2fc750875e0c110b56d2f5b716d4e30b7b"} Oct 29 23:44:03.074088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.074020 5736 scope.go:122] "RemoveContainer" containerID="dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48" Oct 29 23:44:03.089161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.089066 5736 scope.go:122] "RemoveContainer" containerID="dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48" Oct 29 23:44:03.089449 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:44:03.089414 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48\": container with ID starting with dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48 not found: ID does not exist" containerID="dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48" Oct 29 23:44:03.089515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.089462 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48"} err="failed to get container status \"dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48\": rpc error: code = NotFound desc = could not find container \"dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48\": container with ID starting with dbda7a7e3cb68f17513ad543115ddeaceb1884854cdea9718df852a863372b48 not found: ID does not exist" Oct 29 23:44:03.089515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.089485 5736 scope.go:122] "RemoveContainer" containerID="7068750864e51e7939aa7b83ce60996ed0f9ce402437b4a0b9b2e271a07bd9e3" Oct 29 23:44:03.141772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.141733 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr"] Oct 29 23:44:03.153227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.153189 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6d9e36/test-model-6d9e36-kserve-84dc5d565c-g9mjr"] Oct 29 23:44:03.178622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.178595 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz"] Oct 29 23:44:03.182937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.182908 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5cd0f2-86c76f69b4-w5smz"] Oct 29 23:44:03.344994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.337204 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl"] Oct 29 23:44:03.344994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:03.337447 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" containerID="cri-o://c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba" gracePeriod=30 Oct 29 23:44:05.006629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:05.006597 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55389672-b516-4422-b514-3ca1355c361a" path="/var/lib/kubelet/pods/55389672-b516-4422-b514-3ca1355c361a/volumes" Oct 29 23:44:05.007089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:05.007073 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b35f2a-9440-48e6-b6a3-52dd58320ed0" path="/var/lib/kubelet/pods/75b35f2a-9440-48e6-b6a3-52dd58320ed0/volumes" Oct 29 23:44:08.641114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.641084 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:44:08.774026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.773992 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-credential-socket\") pod \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " Oct 29 23:44:08.774223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774035 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-kube-api-access-ncbrp\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-kube-api-access-ncbrp\") pod \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " Oct 29 23:44:08.774223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774103 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-socket\") pod \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " Oct 29 23:44:08.774223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774160 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-envoy\") pod \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " Oct 29 23:44:08.774414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774253 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-credential-socket" pod "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" (UID: "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:08.774414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774265 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istiod-ca-cert\") pod \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " Oct 29 23:44:08.774414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774370 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-token\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-token\") pod \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " Oct 29 23:44:08.774595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774496 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-socket" pod "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" (UID: "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:08.774595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774539 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-data\") pod \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " Oct 29 23:44:08.774700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774605 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istiod-ca-cert" pod "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" (UID: "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:08.774700 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774641 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-podinfo\") pod \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " Oct 29 23:44:08.774813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774706 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-certs\") pod \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\" (UID: \"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8\") " Oct 29 23:44:08.774813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774776 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-data" pod "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" (UID: "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:08.775011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.774955 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-certs" pod "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" (UID: "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:08.775766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.775740 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:08.775897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.775769 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:08.775897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.775784 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:08.775897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.775797 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:08.775897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.775812 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:08.776640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.776613 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-token" pod "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" (UID: "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:08.776745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.776629 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-envoy" pod "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" (UID: "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:08.776745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.776657 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-kube-api-access-ncbrp" pod "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" (UID: "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8"). InnerVolumeSpecName "kube-api-access-ncbrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:08.777047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.777027 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-podinfo" pod "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" (UID: "62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:44:08.877852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.877772 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncbrp\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-kube-api-access-ncbrp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:08.877852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.877800 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:08.877852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.877810 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:08.877852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:08.877820 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:09.180930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:09.180855 5736 generic.go:356] "Generic (PLEG): container finished" podID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerID="c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba" exitCode=0 Oct 29 23:44:09.180930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:09.180900 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" Oct 29 23:44:09.180930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:09.180921 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" event={"ID":"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8","Type":"ContainerDied","Data":"c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba"} Oct 29 23:44:09.181198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:09.180964 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl" event={"ID":"62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8","Type":"ContainerDied","Data":"08c41c04faed863873bd415326603b5c76b3305a2eb6bbd468158b64aaf6413c"} Oct 29 23:44:09.181198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:09.180985 5736 scope.go:122] "RemoveContainer" containerID="c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba" Oct 29 23:44:09.194760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:09.194731 5736 scope.go:122] "RemoveContainer" containerID="c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba" Oct 29 23:44:09.195054 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:44:09.195034 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba\": container with ID starting with c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba not found: ID does not exist" containerID="c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba" Oct 29 23:44:09.195144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:09.195060 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba"} err="failed to get container status \"c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba\": rpc error: code = NotFound desc = could not find container \"c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba\": container with ID starting with c44267c89680357e7dfc83ec7922602d7cb81e65ebc1968e8722c4d839d6a9ba not found: ID does not exist" Oct 29 23:44:09.250321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:09.250295 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl"] Oct 29 23:44:09.261187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:09.261149 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c83b50-openshift-default-587dd5878c-ntzhl"] Oct 29 23:44:10.579434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:10.579398 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45"] Oct 29 23:44:10.639429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:10.637835 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a8ab3c-4hv45"] Oct 29 23:44:11.007150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:11.007113 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" path="/var/lib/kubelet/pods/62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8/volumes" Oct 29 23:44:11.007745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:11.007720 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66753b21-2e82-40e9-a9fe-ae6cc31995a9" path="/var/lib/kubelet/pods/66753b21-2e82-40e9-a9fe-ae6cc31995a9/volumes" Oct 29 23:44:13.966604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:13.965336 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls"] Oct 29 23:44:13.988602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:13.973653 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" podUID="5ab9da55-34af-4cde-8c11-5427d6e81393" containerName="maas-api" containerID="cri-o://798fe8d02555147af8e4e2abab4af9ea39c39679106a87b77259cf1eb44f9037" gracePeriod=30 Oct 29 23:44:14.240705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:14.240602 5736 generic.go:356] "Generic (PLEG): container finished" podID="5ab9da55-34af-4cde-8c11-5427d6e81393" containerID="798fe8d02555147af8e4e2abab4af9ea39c39679106a87b77259cf1eb44f9037" exitCode=0 Oct 29 23:44:14.240930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:14.240764 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" event={"ID":"5ab9da55-34af-4cde-8c11-5427d6e81393","Type":"ContainerDied","Data":"798fe8d02555147af8e4e2abab4af9ea39c39679106a87b77259cf1eb44f9037"} Oct 29 23:44:15.028288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.028255 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:44:15.118220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.118183 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab9da55-34af-4cde-8c11-5427d6e81393-kube-api-access-p6jk5\" (UniqueName: \"kubernetes.io/projected/5ab9da55-34af-4cde-8c11-5427d6e81393-kube-api-access-p6jk5\") pod \"5ab9da55-34af-4cde-8c11-5427d6e81393\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " Oct 29 23:44:15.118411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.118284 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab9da55-34af-4cde-8c11-5427d6e81393-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9da55-34af-4cde-8c11-5427d6e81393-openshift-service-ca\") pod \"5ab9da55-34af-4cde-8c11-5427d6e81393\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " Oct 29 23:44:15.118411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.118377 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls\") pod \"5ab9da55-34af-4cde-8c11-5427d6e81393\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " Oct 29 23:44:15.118702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.118678 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-metrics-certs\") pod \"5ab9da55-34af-4cde-8c11-5427d6e81393\" (UID: \"5ab9da55-34af-4cde-8c11-5427d6e81393\") " Oct 29 23:44:15.118979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.118819 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab9da55-34af-4cde-8c11-5427d6e81393-openshift-service-ca" pod "5ab9da55-34af-4cde-8c11-5427d6e81393" (UID: "5ab9da55-34af-4cde-8c11-5427d6e81393"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:15.119931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.119902 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9da55-34af-4cde-8c11-5427d6e81393-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:15.123292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.123259 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls" pod "5ab9da55-34af-4cde-8c11-5427d6e81393" (UID: "5ab9da55-34af-4cde-8c11-5427d6e81393"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:15.123748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.123717 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab9da55-34af-4cde-8c11-5427d6e81393-kube-api-access-p6jk5" pod "5ab9da55-34af-4cde-8c11-5427d6e81393" (UID: "5ab9da55-34af-4cde-8c11-5427d6e81393"). InnerVolumeSpecName "kube-api-access-p6jk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:15.123938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.123917 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-metrics-certs" pod "5ab9da55-34af-4cde-8c11-5427d6e81393" (UID: "5ab9da55-34af-4cde-8c11-5427d6e81393"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:15.194757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.194677 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs"] Oct 29 23:44:15.197360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.197334 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cd0f2-pslcs"] Oct 29 23:44:15.221997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.221969 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:15.221997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.221994 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6jk5\" (UniqueName: \"kubernetes.io/projected/5ab9da55-34af-4cde-8c11-5427d6e81393-kube-api-access-p6jk5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:15.222147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.222005 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab9da55-34af-4cde-8c11-5427d6e81393-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:15.250418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.250385 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" Oct 29 23:44:15.250418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.250397 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls" event={"ID":"5ab9da55-34af-4cde-8c11-5427d6e81393","Type":"ContainerDied","Data":"7d4aef634436263446b4ad5feeff91b4eba143d56a18dff1555ac6d2f8894083"} Oct 29 23:44:15.250652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.250455 5736 scope.go:122] "RemoveContainer" containerID="798fe8d02555147af8e4e2abab4af9ea39c39679106a87b77259cf1eb44f9037" Oct 29 23:44:15.325506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.325463 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls"] Oct 29 23:44:15.328469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:15.328441 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5cd0f2-5cd696bd9d-9whls"] Oct 29 23:44:17.023945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:17.023650 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8e9925-6fef-4111-be0c-b6bd3147830d" path="/var/lib/kubelet/pods/4b8e9925-6fef-4111-be0c-b6bd3147830d/volumes" Oct 29 23:44:17.025504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:17.025477 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab9da55-34af-4cde-8c11-5427d6e81393" path="/var/lib/kubelet/pods/5ab9da55-34af-4cde-8c11-5427d6e81393/volumes" Oct 29 23:44:19.834574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.834537 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk"] Oct 29 23:44:19.835710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835663 5736 cpu_manager.go:414] "containerMap: removing container" podUID="75b35f2a-9440-48e6-b6a3-52dd58320ed0" containerName="payload-processing" Oct 29 23:44:19.835710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835689 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b35f2a-9440-48e6-b6a3-52dd58320ed0" containerName="payload-processing" Oct 29 23:44:19.835944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835717 5736 cpu_manager.go:414] "containerMap: removing container" podUID="55389672-b516-4422-b514-3ca1355c361a" containerName="main" Oct 29 23:44:19.835944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835725 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55389672-b516-4422-b514-3ca1355c361a" containerName="main" Oct 29 23:44:19.835944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835747 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4b8e9925-6fef-4111-be0c-b6bd3147830d" containerName="revoke-keys" Oct 29 23:44:19.835944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835783 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8e9925-6fef-4111-be0c-b6bd3147830d" containerName="revoke-keys" Oct 29 23:44:19.835944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835837 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5ab9da55-34af-4cde-8c11-5427d6e81393" containerName="maas-api" Oct 29 23:44:19.835944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835846 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab9da55-34af-4cde-8c11-5427d6e81393" containerName="maas-api" Oct 29 23:44:19.835944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835859 5736 cpu_manager.go:414] "containerMap: removing container" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" Oct 29 23:44:19.835944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.835864 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" Oct 29 23:44:19.836379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.836009 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="55389672-b516-4422-b514-3ca1355c361a" containerName="main" Oct 29 23:44:19.836379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.836020 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8e9925-6fef-4111-be0c-b6bd3147830d" containerName="revoke-keys" Oct 29 23:44:19.836379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.836033 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab9da55-34af-4cde-8c11-5427d6e81393" containerName="maas-api" Oct 29 23:44:19.836379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.836042 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d0f97b-8df7-4c65-940e-bbf2c5c1e6a8" containerName="istio-proxy" Oct 29 23:44:19.836379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.836053 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b35f2a-9440-48e6-b6a3-52dd58320ed0" containerName="payload-processing" Oct 29 23:44:19.841720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.841690 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:19.868904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.868873 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d20345\"/\"kube-root-ca.crt\"" Oct 29 23:44:19.873171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.870219 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d20345\"/\"default-dockercfg-gbmnq\"" Oct 29 23:44:19.873171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.870329 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d20345\"/\"openshift-service-ca.crt\"" Oct 29 23:44:19.873171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.870408 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d20345\"/\"test-model-d20345-kserve-self-signed-certs\"" Oct 29 23:44:19.879783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.879749 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk"] Oct 29 23:44:19.986346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.986306 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tmp-dir\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:19.986489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.986375 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-model-cache\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:19.986489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.986448 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgm67\" (UniqueName: \"kubernetes.io/projected/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-kube-api-access-hgm67\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:19.986723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.986703 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tls-certs\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:19.986857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.986835 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-dshm\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:19.986909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:19.986859 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-home\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.106670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.095818 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-model-cache\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.106670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.095882 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgm67\" (UniqueName: \"kubernetes.io/projected/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-kube-api-access-hgm67\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.106670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.096138 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tls-certs\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.106670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.096340 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-dshm\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.106670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.096373 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-home\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.106670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.096573 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tmp-dir\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.106670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.097118 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-model-cache\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.106670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.098710 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tmp-dir\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.106670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.099183 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-home\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.137132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.137095 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgm67\" (UniqueName: \"kubernetes.io/projected/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-kube-api-access-hgm67\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.145037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.144484 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tls-certs\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.146923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.146860 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-dshm\") pod \"test-model-d20345-kserve-6bf7495cf5-4fsbk\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.165009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.164979 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:20.647127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:20.647097 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk"] Oct 29 23:44:20.649775 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:44:20.649743 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7dc7648_76f9_4d85_81ea_ece60b3f76ae.slice/crio-b5e9d2751ab32aa38926d64862c049acfb8f74eaa3edc9fa6370b9ac3778e707 WatchSource:0}: Error finding container b5e9d2751ab32aa38926d64862c049acfb8f74eaa3edc9fa6370b9ac3778e707: Status 404 returned error can't find the container with id b5e9d2751ab32aa38926d64862c049acfb8f74eaa3edc9fa6370b9ac3778e707 Oct 29 23:44:21.342946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:21.342908 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" event={"ID":"f7dc7648-76f9-4d85-81ea-ece60b3f76ae","Type":"ContainerStarted","Data":"00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4"} Oct 29 23:44:21.343445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:21.342960 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" event={"ID":"f7dc7648-76f9-4d85-81ea-ece60b3f76ae","Type":"ContainerStarted","Data":"b5e9d2751ab32aa38926d64862c049acfb8f74eaa3edc9fa6370b9ac3778e707"} Oct 29 23:44:21.387449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:21.387397 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" podStartSLOduration=2.3873828599999998 podStartE2EDuration="2.38738286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:21.383779242 +0000 UTC m=+3658.499698847" watchObservedRunningTime="2026-10-29 23:44:21.38738286 +0000 UTC m=+3658.503302505" Oct 29 23:44:22.166278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:22.166246 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:23.368242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.368209 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5"] Oct 29 23:44:23.374007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.373986 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.395419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.394359 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5"] Oct 29 23:44:23.404848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.404820 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:44:23.404975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.404939 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:44:23.409114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.407996 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmm8\" (UniqueName: \"kubernetes.io/projected/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-kube-api-access-ktmm8\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.409114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.408195 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.409114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.408231 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.510105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.510059 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.510310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.510120 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.510410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.510389 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmm8\" (UniqueName: \"kubernetes.io/projected/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-kube-api-access-ktmm8\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.510816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.510789 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.512591 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.512550 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.552767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.552731 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmm8\" (UniqueName: \"kubernetes.io/projected/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-kube-api-access-ktmm8\") pod \"maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:23.689500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:23.689416 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:24.010464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:24.010430 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5"] Oct 29 23:44:24.012319 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:44:24.012288 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec930e2_cca2_4794_a2ea_fb4b2aa01e75.slice/crio-c2120ea0a06f5b5be7fb0115fd5d53837814a5cc927b87ecb28ed403ab2ac079 WatchSource:0}: Error finding container c2120ea0a06f5b5be7fb0115fd5d53837814a5cc927b87ecb28ed403ab2ac079: Status 404 returned error can't find the container with id c2120ea0a06f5b5be7fb0115fd5d53837814a5cc927b87ecb28ed403ab2ac079 Oct 29 23:44:24.375143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:24.375060 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" event={"ID":"fec930e2-cca2-4794-a2ea-fb4b2aa01e75","Type":"ContainerStarted","Data":"c2120ea0a06f5b5be7fb0115fd5d53837814a5cc927b87ecb28ed403ab2ac079"} Oct 29 23:44:25.385418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:25.385372 5736 generic.go:356] "Generic (PLEG): container finished" podID="fec930e2-cca2-4794-a2ea-fb4b2aa01e75" containerID="88b7d712a3c624a91dd08d9f90939f5ad9315373aa21add0d67bf11f650973e5" exitCode=0 Oct 29 23:44:25.385864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:25.385472 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" event={"ID":"fec930e2-cca2-4794-a2ea-fb4b2aa01e75","Type":"ContainerDied","Data":"88b7d712a3c624a91dd08d9f90939f5ad9315373aa21add0d67bf11f650973e5"} Oct 29 23:44:26.576113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.576083 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:26.680130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.680047 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-openshift-ca\") pod \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " Oct 29 23:44:26.681309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.681276 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-kube-api-access-ktmm8\" (UniqueName: \"kubernetes.io/projected/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-kube-api-access-ktmm8\") pod \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " Oct 29 23:44:26.681794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.681278 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-openshift-ca" pod "fec930e2-cca2-4794-a2ea-fb4b2aa01e75" (UID: "fec930e2-cca2-4794-a2ea-fb4b2aa01e75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:26.682004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.681971 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-xks-ca\") pod \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\" (UID: \"fec930e2-cca2-4794-a2ea-fb4b2aa01e75\") " Oct 29 23:44:26.683476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.683449 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:26.685275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.685252 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-xks-ca" pod "fec930e2-cca2-4794-a2ea-fb4b2aa01e75" (UID: "fec930e2-cca2-4794-a2ea-fb4b2aa01e75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:26.685988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.685947 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-kube-api-access-ktmm8" pod "fec930e2-cca2-4794-a2ea-fb4b2aa01e75" (UID: "fec930e2-cca2-4794-a2ea-fb4b2aa01e75"). InnerVolumeSpecName "kube-api-access-ktmm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:26.784679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.784648 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktmm8\" (UniqueName: \"kubernetes.io/projected/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-kube-api-access-ktmm8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:26.784679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:26.784673 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec930e2-cca2-4794-a2ea-fb4b2aa01e75-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:26.825653 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:44:26.825619 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:44:27.412011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:27.411921 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" event={"ID":"fec930e2-cca2-4794-a2ea-fb4b2aa01e75","Type":"ContainerDied","Data":"c2120ea0a06f5b5be7fb0115fd5d53837814a5cc927b87ecb28ed403ab2ac079"} Oct 29 23:44:27.412254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:27.412237 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2120ea0a06f5b5be7fb0115fd5d53837814a5cc927b87ecb28ed403ab2ac079" Oct 29 23:44:27.412398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:27.412371 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5" Oct 29 23:44:27.507480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:27.507443 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht"] Oct 29 23:44:27.507820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:27.507768 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" containerID="cri-o://63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b" gracePeriod=30 Oct 29 23:44:28.723299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.723261 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7"] Oct 29 23:44:28.724083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.724065 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fec930e2-cca2-4794-a2ea-fb4b2aa01e75" containerName="revoke-keys" Oct 29 23:44:28.724144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.724085 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec930e2-cca2-4794-a2ea-fb4b2aa01e75" containerName="revoke-keys" Oct 29 23:44:28.724258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.724248 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec930e2-cca2-4794-a2ea-fb4b2aa01e75" containerName="revoke-keys" Oct 29 23:44:28.728168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.728150 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:28.749007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.748980 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7"] Oct 29 23:44:28.756318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.756295 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:44:28.756656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.756603 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:44:28.879068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.879037 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682cb002-zw8x7\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:28.879304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.879074 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwfp\" (UniqueName: \"kubernetes.io/projected/70dcfde7-dc28-4c55-890d-ac822e0886a2-kube-api-access-5kwfp\") pod \"maas-api-revoke-sub-682cb002-zw8x7\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:28.879304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.879275 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682cb002-zw8x7\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:28.980903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.980818 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682cb002-zw8x7\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:28.981081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.981057 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwfp\" (UniqueName: \"kubernetes.io/projected/70dcfde7-dc28-4c55-890d-ac822e0886a2-kube-api-access-5kwfp\") pod \"maas-api-revoke-sub-682cb002-zw8x7\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:28.981236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.981220 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682cb002-zw8x7\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:28.981844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.981824 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682cb002-zw8x7\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:28.983210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:28.983182 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682cb002-zw8x7\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:29.005771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:29.005742 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwfp\" (UniqueName: \"kubernetes.io/projected/70dcfde7-dc28-4c55-890d-ac822e0886a2-kube-api-access-5kwfp\") pod \"maas-api-revoke-sub-682cb002-zw8x7\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:29.042728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:29.042697 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:29.379739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:29.379659 5736 scope.go:122] "RemoveContainer" containerID="d1617e1371a408a7059b6fbfab6645bd4d811a0c4e972318292c42deb048f11e" Oct 29 23:44:29.419434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:29.419404 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7"] Oct 29 23:44:29.435075 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:44:29.435046 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dcfde7_dc28_4c55_890d_ac822e0886a2.slice/crio-2a5b2d232930b4c66a49990c1fd1e675ddec7e098802489cf03cc5110bd069d3 WatchSource:0}: Error finding container 2a5b2d232930b4c66a49990c1fd1e675ddec7e098802489cf03cc5110bd069d3: Status 404 returned error can't find the container with id 2a5b2d232930b4c66a49990c1fd1e675ddec7e098802489cf03cc5110bd069d3 Oct 29 23:44:30.166312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:30.166270 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:30.178853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:30.178825 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:30.449383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:30.449296 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" event={"ID":"70dcfde7-dc28-4c55-890d-ac822e0886a2","Type":"ContainerStarted","Data":"0597373c2cdad28c9b8abf8e90f29b0e6f94e718a6e9ddbb40ee6a836706a5c3"} Oct 29 23:44:30.449383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:30.449360 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" event={"ID":"70dcfde7-dc28-4c55-890d-ac822e0886a2","Type":"ContainerStarted","Data":"2a5b2d232930b4c66a49990c1fd1e675ddec7e098802489cf03cc5110bd069d3"} Oct 29 23:44:30.461562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:30.461536 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:44:30.501444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:30.501385 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" podStartSLOduration=2.501373273 podStartE2EDuration="2.501373273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:30.488405637 +0000 UTC m=+3667.604325254" watchObservedRunningTime="2026-10-29 23:44:30.501373273 +0000 UTC m=+3667.617292878" Oct 29 23:44:31.462538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:31.462494 5736 generic.go:356] "Generic (PLEG): container finished" podID="70dcfde7-dc28-4c55-890d-ac822e0886a2" containerID="0597373c2cdad28c9b8abf8e90f29b0e6f94e718a6e9ddbb40ee6a836706a5c3" exitCode=0 Oct 29 23:44:31.463061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:31.462572 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" event={"ID":"70dcfde7-dc28-4c55-890d-ac822e0886a2","Type":"ContainerDied","Data":"0597373c2cdad28c9b8abf8e90f29b0e6f94e718a6e9ddbb40ee6a836706a5c3"} Oct 29 23:44:32.808320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:32.807793 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:32.915647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:32.912139 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-xks-ca\") pod \"70dcfde7-dc28-4c55-890d-ac822e0886a2\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " Oct 29 23:44:32.915647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:32.912235 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70dcfde7-dc28-4c55-890d-ac822e0886a2-kube-api-access-5kwfp\" (UniqueName: \"kubernetes.io/projected/70dcfde7-dc28-4c55-890d-ac822e0886a2-kube-api-access-5kwfp\") pod \"70dcfde7-dc28-4c55-890d-ac822e0886a2\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " Oct 29 23:44:32.915647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:32.912302 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-openshift-ca\") pod \"70dcfde7-dc28-4c55-890d-ac822e0886a2\" (UID: \"70dcfde7-dc28-4c55-890d-ac822e0886a2\") " Oct 29 23:44:32.926741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:32.920116 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-openshift-ca" pod "70dcfde7-dc28-4c55-890d-ac822e0886a2" (UID: "70dcfde7-dc28-4c55-890d-ac822e0886a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:33.120665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.119056 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.282063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.282006 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-xks-ca" pod "70dcfde7-dc28-4c55-890d-ac822e0886a2" (UID: "70dcfde7-dc28-4c55-890d-ac822e0886a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:33.335367 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.335098 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dcfde7-dc28-4c55-890d-ac822e0886a2-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.361710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.361675 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dcfde7-dc28-4c55-890d-ac822e0886a2-kube-api-access-5kwfp" pod "70dcfde7-dc28-4c55-890d-ac822e0886a2" (UID: "70dcfde7-dc28-4c55-890d-ac822e0886a2"). InnerVolumeSpecName "kube-api-access-5kwfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:33.419602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.417023 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:44:33.440189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.440153 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-certs\") pod \"5d529cec-33d9-44f2-b1a6-56dcba642a03\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " Oct 29 23:44:33.440405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.440257 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-data\") pod \"5d529cec-33d9-44f2-b1a6-56dcba642a03\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " Oct 29 23:44:33.440405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.440295 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d529cec-33d9-44f2-b1a6-56dcba642a03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d529cec-33d9-44f2-b1a6-56dcba642a03-istiod-ca-cert\") pod \"5d529cec-33d9-44f2-b1a6-56dcba642a03\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " Oct 29 23:44:33.440537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.440416 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-envoy\") pod \"5d529cec-33d9-44f2-b1a6-56dcba642a03\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " Oct 29 23:44:33.440537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.440472 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-socket\") pod \"5d529cec-33d9-44f2-b1a6-56dcba642a03\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " Oct 29 23:44:33.440731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.440534 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-kube-api-access-str5n\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-kube-api-access-str5n\") pod \"5d529cec-33d9-44f2-b1a6-56dcba642a03\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " Oct 29 23:44:33.440731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.440661 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-token\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-token\") pod \"5d529cec-33d9-44f2-b1a6-56dcba642a03\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " Oct 29 23:44:33.440848 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.440786 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-podinfo\") pod \"5d529cec-33d9-44f2-b1a6-56dcba642a03\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " Oct 29 23:44:33.440908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.440895 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-credential-socket\") pod \"5d529cec-33d9-44f2-b1a6-56dcba642a03\" (UID: \"5d529cec-33d9-44f2-b1a6-56dcba642a03\") " Oct 29 23:44:33.441674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.441644 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kwfp\" (UniqueName: \"kubernetes.io/projected/70dcfde7-dc28-4c55-890d-ac822e0886a2-kube-api-access-5kwfp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.441820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.441717 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-data" pod "5d529cec-33d9-44f2-b1a6-56dcba642a03" (UID: "5d529cec-33d9-44f2-b1a6-56dcba642a03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:33.441985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.441962 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-certs" pod "5d529cec-33d9-44f2-b1a6-56dcba642a03" (UID: "5d529cec-33d9-44f2-b1a6-56dcba642a03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:33.442061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.441988 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d529cec-33d9-44f2-b1a6-56dcba642a03-istiod-ca-cert" pod "5d529cec-33d9-44f2-b1a6-56dcba642a03" (UID: "5d529cec-33d9-44f2-b1a6-56dcba642a03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:33.442483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.442458 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-credential-socket" pod "5d529cec-33d9-44f2-b1a6-56dcba642a03" (UID: "5d529cec-33d9-44f2-b1a6-56dcba642a03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:33.442767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.442744 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-socket" pod "5d529cec-33d9-44f2-b1a6-56dcba642a03" (UID: "5d529cec-33d9-44f2-b1a6-56dcba642a03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:33.444322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.444299 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-podinfo" pod "5d529cec-33d9-44f2-b1a6-56dcba642a03" (UID: "5d529cec-33d9-44f2-b1a6-56dcba642a03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:44:33.444573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.444549 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-envoy" pod "5d529cec-33d9-44f2-b1a6-56dcba642a03" (UID: "5d529cec-33d9-44f2-b1a6-56dcba642a03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:33.444757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.444707 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-kube-api-access-str5n" pod "5d529cec-33d9-44f2-b1a6-56dcba642a03" (UID: "5d529cec-33d9-44f2-b1a6-56dcba642a03"). InnerVolumeSpecName "kube-api-access-str5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:33.445200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.445165 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-token" pod "5d529cec-33d9-44f2-b1a6-56dcba642a03" (UID: "5d529cec-33d9-44f2-b1a6-56dcba642a03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:33.514716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.514664 5736 generic.go:356] "Generic (PLEG): container finished" podID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerID="63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b" exitCode=0 Oct 29 23:44:33.514914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.514716 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" Oct 29 23:44:33.514914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.514737 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" event={"ID":"5d529cec-33d9-44f2-b1a6-56dcba642a03","Type":"ContainerDied","Data":"63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b"} Oct 29 23:44:33.514914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.514812 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht" event={"ID":"5d529cec-33d9-44f2-b1a6-56dcba642a03","Type":"ContainerDied","Data":"1542428581ca2e1d2fe35fe343d842a818afdbc45ed7b0f35fde7b5b55e807f9"} Oct 29 23:44:33.514914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.514846 5736 scope.go:122] "RemoveContainer" containerID="63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b" Oct 29 23:44:33.518152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.518127 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" event={"ID":"70dcfde7-dc28-4c55-890d-ac822e0886a2","Type":"ContainerDied","Data":"2a5b2d232930b4c66a49990c1fd1e675ddec7e098802489cf03cc5110bd069d3"} Oct 29 23:44:33.518459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.518440 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a5b2d232930b4c66a49990c1fd1e675ddec7e098802489cf03cc5110bd069d3" Oct 29 23:44:33.518639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.518400 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7" Oct 29 23:44:33.538312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.537855 5736 scope.go:122] "RemoveContainer" containerID="d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8" Oct 29 23:44:33.543268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.543240 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.543382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.543275 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d529cec-33d9-44f2-b1a6-56dcba642a03-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.543382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.543290 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.543382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.543306 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.543382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.543320 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-str5n\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-kube-api-access-str5n\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.543382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.543335 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.543382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.543349 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d529cec-33d9-44f2-b1a6-56dcba642a03-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.543382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.543363 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.543382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.543376 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d529cec-33d9-44f2-b1a6-56dcba642a03-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:33.550714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.550695 5736 scope.go:122] "RemoveContainer" containerID="63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b" Oct 29 23:44:33.551000 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:44:33.550980 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b\": container with ID starting with 63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b not found: ID does not exist" containerID="63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b" Oct 29 23:44:33.551045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.551009 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b"} err="failed to get container status \"63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b\": rpc error: code = NotFound desc = could not find container \"63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b\": container with ID starting with 63cb7c78f80effd19d84eadd347bd5193b0201165ab9deac2cf8966a16ca032b not found: ID does not exist" Oct 29 23:44:33.551090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.551045 5736 scope.go:122] "RemoveContainer" containerID="d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8" Oct 29 23:44:33.551339 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:44:33.551312 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8\": container with ID starting with d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8 not found: ID does not exist" containerID="d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8" Oct 29 23:44:33.551455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.551343 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8"} err="failed to get container status \"d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8\": rpc error: code = NotFound desc = could not find container \"d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8\": container with ID starting with d6e8ee9ddc1b6d0d2badfc953dd6bae70778cb3ab20c4921b172b6141634d1a8 not found: ID does not exist" Oct 29 23:44:33.573356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.573333 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht"] Oct 29 23:44:33.579090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.579067 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm"] Oct 29 23:44:33.600613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.597010 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a8ab3c-openshift-default-786d89bd7-5ztht"] Oct 29 23:44:33.624543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:33.623234 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5e482-xtjbm"] Oct 29 23:44:33.632657 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:44:33.631700 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dcfde7_dc28_4c55_890d_ac822e0886a2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d529cec_33d9_44f2_b1a6_56dcba642a03.slice/crio-1542428581ca2e1d2fe35fe343d842a818afdbc45ed7b0f35fde7b5b55e807f9\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d529cec_33d9_44f2_b1a6_56dcba642a03.slice\": RecentStats: unable to find data in memory cache]" Oct 29 23:44:35.006410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:35.006375 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2832935d-1204-4b0d-b898-30ac03a097d6" path="/var/lib/kubelet/pods/2832935d-1204-4b0d-b898-30ac03a097d6/volumes" Oct 29 23:44:35.006952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:35.006933 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" path="/var/lib/kubelet/pods/5d529cec-33d9-44f2-b1a6-56dcba642a03/volumes" Oct 29 23:44:36.691781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:36.691747 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5"] Oct 29 23:44:36.701296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:36.701264 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6d9e36-vxdr5"] Oct 29 23:44:37.007834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:37.007798 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec930e2-cca2-4794-a2ea-fb4b2aa01e75" path="/var/lib/kubelet/pods/fec930e2-cca2-4794-a2ea-fb4b2aa01e75/volumes" Oct 29 23:44:38.112455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.112419 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm"] Oct 29 23:44:38.113201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.113170 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" Oct 29 23:44:38.113201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.113201 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" Oct 29 23:44:38.113371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.113232 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" Oct 29 23:44:38.113371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.113238 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" Oct 29 23:44:38.113371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.113261 5736 cpu_manager.go:414] "containerMap: removing container" podUID="70dcfde7-dc28-4c55-890d-ac822e0886a2" containerName="revoke-keys" Oct 29 23:44:38.113371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.113267 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dcfde7-dc28-4c55-890d-ac822e0886a2" containerName="revoke-keys" Oct 29 23:44:38.113505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.113405 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" Oct 29 23:44:38.113505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.113417 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d529cec-33d9-44f2-b1a6-56dcba642a03" containerName="istio-proxy" Oct 29 23:44:38.113505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.113428 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="70dcfde7-dc28-4c55-890d-ac822e0886a2" containerName="revoke-keys" Oct 29 23:44:38.118883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.118864 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.139729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.139693 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm"] Oct 29 23:44:38.146544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.146506 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:44:38.146716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.146576 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:44:38.273905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.273872 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.274124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.274092 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.274320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.274293 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2cc\" (UniqueName: \"kubernetes.io/projected/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-kube-api-access-5r2cc\") pod \"maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.377316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.375982 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.377316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.376274 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.377316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.376426 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2cc\" (UniqueName: \"kubernetes.io/projected/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-kube-api-access-5r2cc\") pod \"maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.377811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.377732 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.384813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.384783 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.406164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.402411 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2cc\" (UniqueName: \"kubernetes.io/projected/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-kube-api-access-5r2cc\") pod \"maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.434230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.434204 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:38.744384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:38.744352 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm"] Oct 29 23:44:38.746617 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:44:38.746545 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef13968_b3ee_4edc_b04b_a2104f8e4c58.slice/crio-48b62b2b9fb1a9528c7975644c2b2191af75f33044e9c66fe193c793a13d0452 WatchSource:0}: Error finding container 48b62b2b9fb1a9528c7975644c2b2191af75f33044e9c66fe193c793a13d0452: Status 404 returned error can't find the container with id 48b62b2b9fb1a9528c7975644c2b2191af75f33044e9c66fe193c793a13d0452 Oct 29 23:44:39.684731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:39.684690 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl"] Oct 29 23:44:39.725827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:39.725785 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" event={"ID":"6ef13968-b3ee-4edc-b04b-a2104f8e4c58","Type":"ContainerStarted","Data":"e53038da1d4b9cdd55a7f432efe241846fa21d27a70be20591bcbe9f9cf982d0"} Oct 29 23:44:39.726002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:39.725856 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" event={"ID":"6ef13968-b3ee-4edc-b04b-a2104f8e4c58","Type":"ContainerStarted","Data":"48b62b2b9fb1a9528c7975644c2b2191af75f33044e9c66fe193c793a13d0452"} Oct 29 23:44:39.848601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:39.842951 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" podStartSLOduration=1.8429326750000001 podStartE2EDuration="1.842932675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:39.804946941 +0000 UTC m=+3676.920866548" watchObservedRunningTime="2026-10-29 23:44:39.842932675 +0000 UTC m=+3676.958852285" Oct 29 23:44:40.372346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.372310 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq"] Oct 29 23:44:40.377175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.377151 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.401241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.401191 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq"] Oct 29 23:44:40.441730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.441690 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dec3212-2v7rq\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.441919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.441790 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96mx\" (UniqueName: \"kubernetes.io/projected/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-kube-api-access-h96mx\") pod \"maas-api-revoke-sub-1dec3212-2v7rq\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.442047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.442023 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dec3212-2v7rq\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.543340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.543310 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96mx\" (UniqueName: \"kubernetes.io/projected/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-kube-api-access-h96mx\") pod \"maas-api-revoke-sub-1dec3212-2v7rq\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.543515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.543489 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dec3212-2v7rq\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.543730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.543708 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dec3212-2v7rq\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.544225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.544199 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dec3212-2v7rq\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.546281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.546260 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dec3212-2v7rq\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.566342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.566319 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96mx\" (UniqueName: \"kubernetes.io/projected/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-kube-api-access-h96mx\") pod \"maas-api-revoke-sub-1dec3212-2v7rq\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.692067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.691998 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:40.737786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.737742 5736 generic.go:356] "Generic (PLEG): container finished" podID="6ef13968-b3ee-4edc-b04b-a2104f8e4c58" containerID="e53038da1d4b9cdd55a7f432efe241846fa21d27a70be20591bcbe9f9cf982d0" exitCode=0 Oct 29 23:44:40.737972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:40.737807 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" event={"ID":"6ef13968-b3ee-4edc-b04b-a2104f8e4c58","Type":"ContainerDied","Data":"e53038da1d4b9cdd55a7f432efe241846fa21d27a70be20591bcbe9f9cf982d0"} Oct 29 23:44:41.017402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.017370 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq"] Oct 29 23:44:41.018339 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:44:41.018309 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cebdb8b_1c50_42bf_a3f6_79a1f5e0eb2a.slice/crio-f3b61738c0208888e6a6e2f7fab6924e1557193f3b053cd5ca0e614129f4f1a5 WatchSource:0}: Error finding container f3b61738c0208888e6a6e2f7fab6924e1557193f3b053cd5ca0e614129f4f1a5: Status 404 returned error can't find the container with id f3b61738c0208888e6a6e2f7fab6924e1557193f3b053cd5ca0e614129f4f1a5 Oct 29 23:44:41.754822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.754529 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" event={"ID":"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a","Type":"ContainerStarted","Data":"6d1122b89cb54481c53f9935fa781d3397f892963e7fabca80c80006c8a4e566"} Oct 29 23:44:41.754822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.754612 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" event={"ID":"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a","Type":"ContainerStarted","Data":"f3b61738c0208888e6a6e2f7fab6924e1557193f3b053cd5ca0e614129f4f1a5"} Oct 29 23:44:41.799253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.799187 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" podStartSLOduration=1.799169018 podStartE2EDuration="1.799169018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:41.794850869 +0000 UTC m=+3678.910770480" watchObservedRunningTime="2026-10-29 23:44:41.799169018 +0000 UTC m=+3678.915088625" Oct 29 23:44:41.958220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.958193 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:41.964454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.964423 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq"] Oct 29 23:44:41.965252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.965230 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6ef13968-b3ee-4edc-b04b-a2104f8e4c58" containerName="revoke-keys" Oct 29 23:44:41.965353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.965252 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef13968-b3ee-4edc-b04b-a2104f8e4c58" containerName="revoke-keys" Oct 29 23:44:41.965497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.965482 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef13968-b3ee-4edc-b04b-a2104f8e4c58" containerName="revoke-keys" Oct 29 23:44:41.969130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.969111 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:41.995716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.995685 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq"] Oct 29 23:44:41.998355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.998328 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-kube-api-access-5r2cc\" (UniqueName: \"kubernetes.io/projected/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-kube-api-access-5r2cc\") pod \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " Oct 29 23:44:41.998494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.998475 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-openshift-ca\") pod \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " Oct 29 23:44:41.998705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.998686 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-xks-ca\") pod \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\" (UID: \"6ef13968-b3ee-4edc-b04b-a2104f8e4c58\") " Oct 29 23:44:41.998940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.998912 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-openshift-ca" pod "6ef13968-b3ee-4edc-b04b-a2104f8e4c58" (UID: "6ef13968-b3ee-4edc-b04b-a2104f8e4c58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:41.999024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.998963 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcfz\" (UniqueName: \"kubernetes.io/projected/d4c71943-9377-4161-bb98-8850f7f14f97-kube-api-access-kgcfz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:41.999358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.999305 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:41.999477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.999448 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:41.999793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:41.999772 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:42.001494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.001468 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-kube-api-access-5r2cc" pod "6ef13968-b3ee-4edc-b04b-a2104f8e4c58" (UID: "6ef13968-b3ee-4edc-b04b-a2104f8e4c58"). InnerVolumeSpecName "kube-api-access-5r2cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:42.001691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.001555 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-xks-ca" pod "6ef13968-b3ee-4edc-b04b-a2104f8e4c58" (UID: "6ef13968-b3ee-4edc-b04b-a2104f8e4c58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:42.100480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.100443 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcfz\" (UniqueName: \"kubernetes.io/projected/d4c71943-9377-4161-bb98-8850f7f14f97-kube-api-access-kgcfz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:42.100685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.100633 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:42.100685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.100677 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:42.100840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.100800 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r2cc\" (UniqueName: \"kubernetes.io/projected/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-kube-api-access-5r2cc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:42.100840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.100817 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef13968-b3ee-4edc-b04b-a2104f8e4c58-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:42.101463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.101436 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:42.103053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.103034 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:42.123713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.123689 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcfz\" (UniqueName: \"kubernetes.io/projected/d4c71943-9377-4161-bb98-8850f7f14f97-kube-api-access-kgcfz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:42.283573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.283540 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:42.593932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.593903 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq"] Oct 29 23:44:42.595929 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:44:42.595896 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c71943_9377_4161_bb98_8850f7f14f97.slice/crio-c636a93f35390417c643f629bc9d721651ef84b3b958542a9e67288f09560d75 WatchSource:0}: Error finding container c636a93f35390417c643f629bc9d721651ef84b3b958542a9e67288f09560d75: Status 404 returned error can't find the container with id c636a93f35390417c643f629bc9d721651ef84b3b958542a9e67288f09560d75 Oct 29 23:44:42.763150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.763122 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" Oct 29 23:44:42.763548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.763146 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm" event={"ID":"6ef13968-b3ee-4edc-b04b-a2104f8e4c58","Type":"ContainerDied","Data":"48b62b2b9fb1a9528c7975644c2b2191af75f33044e9c66fe193c793a13d0452"} Oct 29 23:44:42.763548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.763215 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48b62b2b9fb1a9528c7975644c2b2191af75f33044e9c66fe193c793a13d0452" Oct 29 23:44:42.764807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.764781 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" event={"ID":"d4c71943-9377-4161-bb98-8850f7f14f97","Type":"ContainerStarted","Data":"c636a93f35390417c643f629bc9d721651ef84b3b958542a9e67288f09560d75"} Oct 29 23:44:42.766858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.766830 5736 generic.go:356] "Generic (PLEG): container finished" podID="6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a" containerID="6d1122b89cb54481c53f9935fa781d3397f892963e7fabca80c80006c8a4e566" exitCode=0 Oct 29 23:44:42.766988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:42.766898 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" event={"ID":"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a","Type":"ContainerDied","Data":"6d1122b89cb54481c53f9935fa781d3397f892963e7fabca80c80006c8a4e566"} Oct 29 23:44:43.781325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:43.781282 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" event={"ID":"d4c71943-9377-4161-bb98-8850f7f14f97","Type":"ContainerStarted","Data":"ab7f2056fb9dcae88a6cccd64c16099511bf347f5bec45164e76da2ef6e2525a"} Oct 29 23:44:43.999601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:43.993071 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" podStartSLOduration=2.9930502629999998 podStartE2EDuration="2.993050263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:43.898215715 +0000 UTC m=+3681.014135322" watchObservedRunningTime="2026-10-29 23:44:43.993050263 +0000 UTC m=+3681.108969870" Oct 29 23:44:44.186906 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:44:44.186866 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c71943_9377_4161_bb98_8850f7f14f97.slice/crio-conmon-ab7f2056fb9dcae88a6cccd64c16099511bf347f5bec45164e76da2ef6e2525a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c71943_9377_4161_bb98_8850f7f14f97.slice/crio-ab7f2056fb9dcae88a6cccd64c16099511bf347f5bec45164e76da2ef6e2525a.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:44:44.254760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.254731 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:44.295348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.295319 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-kube-api-access-h96mx\" (UniqueName: \"kubernetes.io/projected/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-kube-api-access-h96mx\") pod \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " Oct 29 23:44:44.295513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.295496 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-openshift-ca\") pod \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " Oct 29 23:44:44.295556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.295542 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-xks-ca\") pod \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\" (UID: \"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a\") " Oct 29 23:44:44.295954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.295921 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-openshift-ca" pod "6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a" (UID: "6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:44.296439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.296418 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:44.297411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.297383 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-kube-api-access-h96mx" pod "6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a" (UID: "6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a"). InnerVolumeSpecName "kube-api-access-h96mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:44.297639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.297612 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-xks-ca" pod "6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a" (UID: "6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:44.333745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.333713 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb"] Oct 29 23:44:44.334968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.334943 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a" containerName="revoke-keys" Oct 29 23:44:44.334968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.334968 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a" containerName="revoke-keys" Oct 29 23:44:44.335186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.335172 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a" containerName="revoke-keys" Oct 29 23:44:44.345490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.339838 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.350543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.350515 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb"] Oct 29 23:44:44.411802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.411726 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd445\" (UniqueName: \"kubernetes.io/projected/94abd8a1-519d-4c4a-8217-4b153bcc17a5-kube-api-access-kd445\") pod \"maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.412116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.412094 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.412189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.412161 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.412352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.412326 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:44.412408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.412359 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h96mx\" (UniqueName: \"kubernetes.io/projected/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a-kube-api-access-h96mx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:44.513386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.513358 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.513557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.513409 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.513557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.513480 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd445\" (UniqueName: \"kubernetes.io/projected/94abd8a1-519d-4c4a-8217-4b153bcc17a5-kube-api-access-kd445\") pod \"maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.514102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.514079 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.515870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.515849 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.536300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.536277 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd445\" (UniqueName: \"kubernetes.io/projected/94abd8a1-519d-4c4a-8217-4b153bcc17a5-kube-api-access-kd445\") pod \"maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.666298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.666219 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:44.797278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.797244 5736 generic.go:356] "Generic (PLEG): container finished" podID="d4c71943-9377-4161-bb98-8850f7f14f97" containerID="ab7f2056fb9dcae88a6cccd64c16099511bf347f5bec45164e76da2ef6e2525a" exitCode=0 Oct 29 23:44:44.797752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.797331 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" event={"ID":"d4c71943-9377-4161-bb98-8850f7f14f97","Type":"ContainerDied","Data":"ab7f2056fb9dcae88a6cccd64c16099511bf347f5bec45164e76da2ef6e2525a"} Oct 29 23:44:44.799063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.799037 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" event={"ID":"6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a","Type":"ContainerDied","Data":"f3b61738c0208888e6a6e2f7fab6924e1557193f3b053cd5ca0e614129f4f1a5"} Oct 29 23:44:44.799201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.799079 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b61738c0208888e6a6e2f7fab6924e1557193f3b053cd5ca0e614129f4f1a5" Oct 29 23:44:44.799201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.799126 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq" Oct 29 23:44:44.976227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:44.976199 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb"] Oct 29 23:44:44.978503 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:44:44.978472 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94abd8a1_519d_4c4a_8217_4b153bcc17a5.slice/crio-cffc5ea76b2f875fc4d3ca693f018046fb3ecaa7a3071b23e724df2ebb277c26 WatchSource:0}: Error finding container cffc5ea76b2f875fc4d3ca693f018046fb3ecaa7a3071b23e724df2ebb277c26: Status 404 returned error can't find the container with id cffc5ea76b2f875fc4d3ca693f018046fb3ecaa7a3071b23e724df2ebb277c26 Oct 29 23:44:45.809040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:45.808951 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" event={"ID":"94abd8a1-519d-4c4a-8217-4b153bcc17a5","Type":"ContainerStarted","Data":"67a452701f2a04310723678c4661970ab11d0d3a99443b2a003a2f00d93bdb54"} Oct 29 23:44:45.809040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:45.809015 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" event={"ID":"94abd8a1-519d-4c4a-8217-4b153bcc17a5","Type":"ContainerStarted","Data":"cffc5ea76b2f875fc4d3ca693f018046fb3ecaa7a3071b23e724df2ebb277c26"} Oct 29 23:44:45.851945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:45.851888 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" podStartSLOduration=1.85187253 podStartE2EDuration="1.85187253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:45.849250826 +0000 UTC m=+3682.965170432" watchObservedRunningTime="2026-10-29 23:44:45.85187253 +0000 UTC m=+3682.967792136" Oct 29 23:44:45.989161 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:45.989133 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:46.041223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.041190 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-xks-ca\") pod \"d4c71943-9377-4161-bb98-8850f7f14f97\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " Oct 29 23:44:46.041396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.041300 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-openshift-ca\") pod \"d4c71943-9377-4161-bb98-8850f7f14f97\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " Oct 29 23:44:46.041451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.041405 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c71943-9377-4161-bb98-8850f7f14f97-kube-api-access-kgcfz\" (UniqueName: \"kubernetes.io/projected/d4c71943-9377-4161-bb98-8850f7f14f97-kube-api-access-kgcfz\") pod \"d4c71943-9377-4161-bb98-8850f7f14f97\" (UID: \"d4c71943-9377-4161-bb98-8850f7f14f97\") " Oct 29 23:44:46.041744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.041716 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-openshift-ca" pod "d4c71943-9377-4161-bb98-8850f7f14f97" (UID: "d4c71943-9377-4161-bb98-8850f7f14f97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:46.042563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.042536 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.043804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.043775 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c71943-9377-4161-bb98-8850f7f14f97-kube-api-access-kgcfz" pod "d4c71943-9377-4161-bb98-8850f7f14f97" (UID: "d4c71943-9377-4161-bb98-8850f7f14f97"). InnerVolumeSpecName "kube-api-access-kgcfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:46.044323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.044302 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-xks-ca" pod "d4c71943-9377-4161-bb98-8850f7f14f97" (UID: "d4c71943-9377-4161-bb98-8850f7f14f97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:46.144026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.143988 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c71943-9377-4161-bb98-8850f7f14f97-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.144026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.144019 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgcfz\" (UniqueName: \"kubernetes.io/projected/d4c71943-9377-4161-bb98-8850f7f14f97-kube-api-access-kgcfz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:46.818819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.818784 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" event={"ID":"d4c71943-9377-4161-bb98-8850f7f14f97","Type":"ContainerDied","Data":"c636a93f35390417c643f629bc9d721651ef84b3b958542a9e67288f09560d75"} Oct 29 23:44:46.818819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.818805 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq" Oct 29 23:44:46.819298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.818843 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c636a93f35390417c643f629bc9d721651ef84b3b958542a9e67288f09560d75" Oct 29 23:44:46.820740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.820710 5736 generic.go:356] "Generic (PLEG): container finished" podID="94abd8a1-519d-4c4a-8217-4b153bcc17a5" containerID="67a452701f2a04310723678c4661970ab11d0d3a99443b2a003a2f00d93bdb54" exitCode=0 Oct 29 23:44:46.820857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:46.820746 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" event={"ID":"94abd8a1-519d-4c4a-8217-4b153bcc17a5","Type":"ContainerDied","Data":"67a452701f2a04310723678c4661970ab11d0d3a99443b2a003a2f00d93bdb54"} Oct 29 23:44:48.003474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.003446 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:48.082657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.082554 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-xks-ca\") pod \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " Oct 29 23:44:48.082823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.082663 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-openshift-ca\") pod \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " Oct 29 23:44:48.082997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.082975 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94abd8a1-519d-4c4a-8217-4b153bcc17a5-kube-api-access-kd445\" (UniqueName: \"kubernetes.io/projected/94abd8a1-519d-4c4a-8217-4b153bcc17a5-kube-api-access-kd445\") pod \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\" (UID: \"94abd8a1-519d-4c4a-8217-4b153bcc17a5\") " Oct 29 23:44:48.083092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.083038 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-openshift-ca" pod "94abd8a1-519d-4c4a-8217-4b153bcc17a5" (UID: "94abd8a1-519d-4c4a-8217-4b153bcc17a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:44:48.084077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.084056 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:48.085448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.085418 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-xks-ca" pod "94abd8a1-519d-4c4a-8217-4b153bcc17a5" (UID: "94abd8a1-519d-4c4a-8217-4b153bcc17a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:48.085766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.085731 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94abd8a1-519d-4c4a-8217-4b153bcc17a5-kube-api-access-kd445" pod "94abd8a1-519d-4c4a-8217-4b153bcc17a5" (UID: "94abd8a1-519d-4c4a-8217-4b153bcc17a5"). InnerVolumeSpecName "kube-api-access-kd445". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:48.185493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.185460 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94abd8a1-519d-4c4a-8217-4b153bcc17a5-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:48.185493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.185488 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd445\" (UniqueName: \"kubernetes.io/projected/94abd8a1-519d-4c4a-8217-4b153bcc17a5-kube-api-access-kd445\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:48.840435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.840404 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" Oct 29 23:44:48.840620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.840431 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb" event={"ID":"94abd8a1-519d-4c4a-8217-4b153bcc17a5","Type":"ContainerDied","Data":"cffc5ea76b2f875fc4d3ca693f018046fb3ecaa7a3071b23e724df2ebb277c26"} Oct 29 23:44:48.840620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:48.840485 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cffc5ea76b2f875fc4d3ca693f018046fb3ecaa7a3071b23e724df2ebb277c26" Oct 29 23:44:54.722999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:54.722905 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" podUID="2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" containerName="main" containerID="cri-o://a61f481ecc61fc7fe137389413ab5b4c1ba8e600ab269f8f5df71e035bc7caae" gracePeriod=45 Oct 29 23:44:54.839159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:54.838773 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" podUID="2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" containerName="main" probeResult="failure" output="Get \"https://10.134.1.108:8000/ready\": dial tcp 10.134.1.108:8000: connect: connection refused" Oct 29 23:44:54.931977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:54.931935 5736 generic.go:356] "Generic (PLEG): container finished" podID="2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" containerID="a61f481ecc61fc7fe137389413ab5b4c1ba8e600ab269f8f5df71e035bc7caae" exitCode=0 Oct 29 23:44:54.932176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:54.931993 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" event={"ID":"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb","Type":"ContainerDied","Data":"a61f481ecc61fc7fe137389413ab5b4c1ba8e600ab269f8f5df71e035bc7caae"} Oct 29 23:44:55.036213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.036187 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:44:55.151102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.151061 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-kube-api-access-xngs5\" (UniqueName: \"kubernetes.io/projected/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-kube-api-access-xngs5\") pod \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " Oct 29 23:44:55.151327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.151122 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-dshm\") pod \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " Oct 29 23:44:55.151327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.151259 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-home\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-home\") pod \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " Oct 29 23:44:55.151327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.151305 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tls-certs\") pod \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " Oct 29 23:44:55.151623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.151377 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-model-cache\") pod \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " Oct 29 23:44:55.152074 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.152013 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tmp-dir\") pod \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\" (UID: \"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb\") " Oct 29 23:44:55.223603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.210102 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-model-cache" pod "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" (UID: "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:55.223603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.210444 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tmp-dir" pod "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" (UID: "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:55.223603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.213086 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-home" pod "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" (UID: "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:55.223603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.216156 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tls-certs" pod "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" (UID: "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:44:55.246738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.246697 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-kube-api-access-xngs5" pod "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" (UID: "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb"). InnerVolumeSpecName "kube-api-access-xngs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:44:55.257050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.257012 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-dshm" pod "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" (UID: "2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:44:55.309625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.307457 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xngs5\" (UniqueName: \"kubernetes.io/projected/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-kube-api-access-xngs5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:55.309625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.307502 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:55.309625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.307518 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:55.309625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.307532 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:55.309625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.307545 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:55.309625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.307557 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:44:55.941745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.941711 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" Oct 29 23:44:55.942212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.941737 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl" event={"ID":"2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb","Type":"ContainerDied","Data":"8e78f09524e68558dcc6f62a8019fc583f35e455e9f8c4953544621431281e41"} Oct 29 23:44:55.942212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:55.941799 5736 scope.go:122] "RemoveContainer" containerID="a61f481ecc61fc7fe137389413ab5b4c1ba8e600ab269f8f5df71e035bc7caae" Oct 29 23:44:56.000512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:56.000475 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl"] Oct 29 23:44:56.011790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:56.011762 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-51462d/test-model-51462d-kserve-6d946b877f-pr9wl"] Oct 29 23:44:56.826014 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:44:56.825975 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:44:57.005747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:44:57.005711 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" path="/var/lib/kubelet/pods/2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb/volumes" Oct 29 23:45:00.148365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.148202 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr"] Oct 29 23:45:00.149411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.149383 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" containerName="main" Oct 29 23:45:00.149411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.149408 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" containerName="main" Oct 29 23:45:00.149729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.149424 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d4c71943-9377-4161-bb98-8850f7f14f97" containerName="revoke-keys" Oct 29 23:45:00.149729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.149431 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c71943-9377-4161-bb98-8850f7f14f97" containerName="revoke-keys" Oct 29 23:45:00.149729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.149439 5736 cpu_manager.go:414] "containerMap: removing container" podUID="94abd8a1-519d-4c4a-8217-4b153bcc17a5" containerName="revoke-keys" Oct 29 23:45:00.149729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.149445 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94abd8a1-519d-4c4a-8217-4b153bcc17a5" containerName="revoke-keys" Oct 29 23:45:00.149729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.149603 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fef1a8e-2143-4c30-955c-1a6ee0cd3bfb" containerName="main" Oct 29 23:45:00.149729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.149615 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c71943-9377-4161-bb98-8850f7f14f97" containerName="revoke-keys" Oct 29 23:45:00.149729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.149621 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="94abd8a1-519d-4c4a-8217-4b153bcc17a5" containerName="revoke-keys" Oct 29 23:45:00.156642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.153573 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:00.156642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.154218 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr"] Oct 29 23:45:00.159803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.159782 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz"] Oct 29 23:45:00.159932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.159908 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:00.164421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.164397 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct"] Oct 29 23:45:00.164563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.164539 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:00.170026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.169995 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:00.170150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.170068 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:45:00.176972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.176733 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr"] Oct 29 23:45:00.183544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.183509 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr"] Oct 29 23:45:00.193860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.193825 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c869l\" (UniqueName: \"kubernetes.io/projected/5a37864a-0480-4fe9-b5c4-c946c1f049a9-kube-api-access-c869l\") pod \"maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr\" (UID: \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:00.194091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.193913 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97knw\" (UniqueName: \"kubernetes.io/projected/74c99f0d-50d0-4226-b5b7-fd1175bedb56-kube-api-access-97knw\") pod \"maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz\" (UID: \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:00.194091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.193947 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c99f0d-50d0-4226-b5b7-fd1175bedb56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz\" (UID: \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:00.194091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.193993 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37864a-0480-4fe9-b5c4-c946c1f049a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr\" (UID: \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:00.194522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.194115 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5fv\" (UniqueName: \"kubernetes.io/projected/137de700-2980-41fc-b778-24aa711d2ac2-kube-api-access-sh5fv\") pod \"maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr\" (UID: \"137de700-2980-41fc-b778-24aa711d2ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:00.194522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.194284 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137de700-2980-41fc-b778-24aa711d2ac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr\" (UID: \"137de700-2980-41fc-b778-24aa711d2ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:00.194837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.194595 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg"] Oct 29 23:45:00.201451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.201420 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz"] Oct 29 23:45:00.201632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.201525 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:00.212653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.208325 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct"] Oct 29 23:45:00.220034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.218796 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth"] Oct 29 23:45:00.231552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.231523 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:00.251433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.248589 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4"] Oct 29 23:45:00.265052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.265023 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:00.267418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.267390 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8"] Oct 29 23:45:00.274324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.274299 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27"] Oct 29 23:45:00.274498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.274478 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:00.291627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.288719 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth"] Oct 29 23:45:00.291627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.288765 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4"] Oct 29 23:45:00.291627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.288792 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg"] Oct 29 23:45:00.291627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.288881 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297085 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9fr\" (UniqueName: \"kubernetes.io/projected/db376677-f21f-4807-ab75-b82ce5e8f51b-kube-api-access-tq9fr\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth\" (UID: \"db376677-f21f-4807-ab75-b82ce5e8f51b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297157 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct\" (UID: \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297240 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137de700-2980-41fc-b778-24aa711d2ac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr\" (UID: \"137de700-2980-41fc-b778-24aa711d2ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297397 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7055786b-a7d7-4894-81c3-65ac4348fea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg\" (UID: \"7055786b-a7d7-4894-81c3-65ac4348fea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297428 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e79f9-8c22-4a9f-87db-bd6413409296-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4\" (UID: \"8c9e79f9-8c22-4a9f-87db-bd6413409296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297481 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c869l\" (UniqueName: \"kubernetes.io/projected/5a37864a-0480-4fe9-b5c4-c946c1f049a9-kube-api-access-c869l\") pod \"maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr\" (UID: \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297549 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jmcc\" (UniqueName: \"kubernetes.io/projected/7055786b-a7d7-4894-81c3-65ac4348fea9-kube-api-access-8jmcc\") pod \"maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg\" (UID: \"7055786b-a7d7-4894-81c3-65ac4348fea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297617 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97knw\" (UniqueName: \"kubernetes.io/projected/74c99f0d-50d0-4226-b5b7-fd1175bedb56-kube-api-access-97knw\") pod \"maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz\" (UID: \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297655 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c99f0d-50d0-4226-b5b7-fd1175bedb56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz\" (UID: \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297707 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37864a-0480-4fe9-b5c4-c946c1f049a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr\" (UID: \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297772 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db376677-f21f-4807-ab75-b82ce5e8f51b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth\" (UID: \"db376677-f21f-4807-ab75-b82ce5e8f51b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297844 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmbr\" (UniqueName: \"kubernetes.io/projected/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-kube-api-access-sgmbr\") pod \"maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct\" (UID: \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297894 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5fv\" (UniqueName: \"kubernetes.io/projected/137de700-2980-41fc-b778-24aa711d2ac2-kube-api-access-sh5fv\") pod \"maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr\" (UID: \"137de700-2980-41fc-b778-24aa711d2ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:00.298833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.297923 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffw9\" (UniqueName: \"kubernetes.io/projected/8c9e79f9-8c22-4a9f-87db-bd6413409296-kube-api-access-vffw9\") pod \"maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4\" (UID: \"8c9e79f9-8c22-4a9f-87db-bd6413409296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:00.299450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.299410 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk"] Oct 29 23:45:00.300090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.300064 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c99f0d-50d0-4226-b5b7-fd1175bedb56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz\" (UID: \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:00.300593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.300560 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37864a-0480-4fe9-b5c4-c946c1f049a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr\" (UID: \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:00.304458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.304430 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137de700-2980-41fc-b778-24aa711d2ac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr\" (UID: \"137de700-2980-41fc-b778-24aa711d2ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:00.313466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.313398 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8"] Oct 29 23:45:00.316351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.316206 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x"] Oct 29 23:45:00.325483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.325458 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:00.349791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.349763 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858"] Oct 29 23:45:00.365537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.350623 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:00.389011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.388970 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c869l\" (UniqueName: \"kubernetes.io/projected/5a37864a-0480-4fe9-b5c4-c946c1f049a9-kube-api-access-c869l\") pod \"maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr\" (UID: \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:00.390055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.389144 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97knw\" (UniqueName: \"kubernetes.io/projected/74c99f0d-50d0-4226-b5b7-fd1175bedb56-kube-api-access-97knw\") pod \"maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz\" (UID: \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:00.395040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.395005 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27"] Oct 29 23:45:00.396315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.396289 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:00.419274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.419180 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59g85\" (UniqueName: \"kubernetes.io/projected/fefefc63-6416-4fec-b277-6a8b9e9de6fc-kube-api-access-59g85\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27\" (UID: \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:00.419469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.419444 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x\" (UID: \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:00.419543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.419527 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefefc63-6416-4fec-b277-6a8b9e9de6fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27\" (UID: \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:00.419656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.419622 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7055786b-a7d7-4894-81c3-65ac4348fea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg\" (UID: \"7055786b-a7d7-4894-81c3-65ac4348fea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:00.419731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.419677 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e79f9-8c22-4a9f-87db-bd6413409296-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4\" (UID: \"8c9e79f9-8c22-4a9f-87db-bd6413409296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:00.419793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.419752 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp665\" (UniqueName: \"kubernetes.io/projected/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-kube-api-access-jp665\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk\" (UID: \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:00.419854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.419841 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmcc\" (UniqueName: \"kubernetes.io/projected/7055786b-a7d7-4894-81c3-65ac4348fea9-kube-api-access-8jmcc\") pod \"maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg\" (UID: \"7055786b-a7d7-4894-81c3-65ac4348fea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:00.420017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.419998 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db376677-f21f-4807-ab75-b82ce5e8f51b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth\" (UID: \"db376677-f21f-4807-ab75-b82ce5e8f51b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:00.420073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.420032 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk\" (UID: \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:00.420134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.420083 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djjl\" (UniqueName: \"kubernetes.io/projected/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-kube-api-access-8djjl\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8\" (UID: \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:00.420201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.420184 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmbr\" (UniqueName: \"kubernetes.io/projected/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-kube-api-access-sgmbr\") pod \"maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct\" (UID: \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:00.420258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.420235 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8\" (UID: \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:00.420325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.420308 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffw9\" (UniqueName: \"kubernetes.io/projected/8c9e79f9-8c22-4a9f-87db-bd6413409296-kube-api-access-vffw9\") pod \"maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4\" (UID: \"8c9e79f9-8c22-4a9f-87db-bd6413409296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:00.420482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.420466 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9fr\" (UniqueName: \"kubernetes.io/projected/db376677-f21f-4807-ab75-b82ce5e8f51b-kube-api-access-tq9fr\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth\" (UID: \"db376677-f21f-4807-ab75-b82ce5e8f51b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:00.420549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.420506 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqtwv\" (UniqueName: \"kubernetes.io/projected/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-kube-api-access-bqtwv\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x\" (UID: \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:00.420744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.420575 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct\" (UID: \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:00.421825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.421799 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct\" (UID: \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:00.422809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.422768 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7055786b-a7d7-4894-81c3-65ac4348fea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg\" (UID: \"7055786b-a7d7-4894-81c3-65ac4348fea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:00.423982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.423945 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e79f9-8c22-4a9f-87db-bd6413409296-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4\" (UID: \"8c9e79f9-8c22-4a9f-87db-bd6413409296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:00.425361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.425339 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db376677-f21f-4807-ab75-b82ce5e8f51b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth\" (UID: \"db376677-f21f-4807-ab75-b82ce5e8f51b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:00.450683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.450651 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk"] Oct 29 23:45:00.451380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.451346 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5fv\" (UniqueName: \"kubernetes.io/projected/137de700-2980-41fc-b778-24aa711d2ac2-kube-api-access-sh5fv\") pod \"maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr\" (UID: \"137de700-2980-41fc-b778-24aa711d2ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:00.468516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.468492 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmcc\" (UniqueName: \"kubernetes.io/projected/7055786b-a7d7-4894-81c3-65ac4348fea9-kube-api-access-8jmcc\") pod \"maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg\" (UID: \"7055786b-a7d7-4894-81c3-65ac4348fea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:00.468704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.468622 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9fr\" (UniqueName: \"kubernetes.io/projected/db376677-f21f-4807-ab75-b82ce5e8f51b-kube-api-access-tq9fr\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth\" (UID: \"db376677-f21f-4807-ab75-b82ce5e8f51b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:00.469280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.469228 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmbr\" (UniqueName: \"kubernetes.io/projected/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-kube-api-access-sgmbr\") pod \"maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct\" (UID: \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:00.469436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.469263 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffw9\" (UniqueName: \"kubernetes.io/projected/8c9e79f9-8c22-4a9f-87db-bd6413409296-kube-api-access-vffw9\") pod \"maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4\" (UID: \"8c9e79f9-8c22-4a9f-87db-bd6413409296\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:00.473658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.471594 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858"] Oct 29 23:45:00.473658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.473016 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:00.482516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.477673 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5"] Oct 29 23:45:00.485479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.485457 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5"] Oct 29 23:45:00.489966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.489803 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:00.490441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.490420 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:00.492932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.492845 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:00.495718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.495695 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4"] Oct 29 23:45:00.504557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.504360 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:00.510705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.510684 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5"] Oct 29 23:45:00.510871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.510858 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x"] Oct 29 23:45:00.510988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.510797 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:00.511187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.511036 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5"] Oct 29 23:45:00.517935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.517910 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj"] Oct 29 23:45:00.525643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.524188 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefefc63-6416-4fec-b277-6a8b9e9de6fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27\" (UID: \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:00.525643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.524256 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp665\" (UniqueName: \"kubernetes.io/projected/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-kube-api-access-jp665\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk\" (UID: \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:00.525643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.524332 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnsdc\" (UniqueName: \"kubernetes.io/projected/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-kube-api-access-xnsdc\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858\" (UID: \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:00.525643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.524370 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk\" (UID: \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:00.525643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.524389 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djjl\" (UniqueName: \"kubernetes.io/projected/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-kube-api-access-8djjl\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8\" (UID: \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:00.525643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.524416 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858\" (UID: \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:00.525643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.524449 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8\" (UID: \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:00.525643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.524524 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqtwv\" (UniqueName: \"kubernetes.io/projected/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-kube-api-access-bqtwv\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x\" (UID: \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:00.525643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.524565 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59g85\" (UniqueName: \"kubernetes.io/projected/fefefc63-6416-4fec-b277-6a8b9e9de6fc-kube-api-access-59g85\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27\" (UID: \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:00.526298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.525821 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk\" (UID: \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:00.527013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.526505 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x\" (UID: \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:00.527722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.527348 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefefc63-6416-4fec-b277-6a8b9e9de6fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27\" (UID: \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:00.527813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.527743 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x\" (UID: \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:00.528120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.527960 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:00.528466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.528399 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8\" (UID: \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:00.533737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.529250 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4"] Oct 29 23:45:00.533737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.530200 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:00.541096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.541057 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4"] Oct 29 23:45:00.541212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.541104 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj"] Oct 29 23:45:00.541263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.541213 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:00.551291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.551086 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djjl\" (UniqueName: \"kubernetes.io/projected/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-kube-api-access-8djjl\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8\" (UID: \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:00.557049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.553007 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp665\" (UniqueName: \"kubernetes.io/projected/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-kube-api-access-jp665\") pod \"maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk\" (UID: \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:00.557049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.554205 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59g85\" (UniqueName: \"kubernetes.io/projected/fefefc63-6416-4fec-b277-6a8b9e9de6fc-kube-api-access-59g85\") pod \"maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27\" (UID: \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:00.563603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.561075 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6"] Oct 29 23:45:00.567429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.567408 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:00.568136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.568059 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4"] Oct 29 23:45:00.570711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.570656 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqtwv\" (UniqueName: \"kubernetes.io/projected/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-kube-api-access-bqtwv\") pod \"maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x\" (UID: \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:00.590272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.584376 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6"] Oct 29 23:45:00.595373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.595341 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:00.601410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.599625 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:00.618418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.618391 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:00.631215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.630319 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnsdc\" (UniqueName: \"kubernetes.io/projected/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-kube-api-access-xnsdc\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858\" (UID: \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:00.631215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.630384 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsftg\" (UniqueName: \"kubernetes.io/projected/aacb0496-adc4-4859-afe2-88a8c07d27b3-kube-api-access-rsftg\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj\" (UID: \"aacb0496-adc4-4859-afe2-88a8c07d27b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:00.631215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.630456 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63cd40-3af1-49cb-b17b-5576a1f434cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4\" (UID: \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:00.631215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.630500 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858\" (UID: \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:00.631215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.630532 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ff5a60-2de6-41b0-b7c9-0864c9086318-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6\" (UID: \"52ff5a60-2de6-41b0-b7c9-0864c9086318\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:00.631215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.630649 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zskqf\" (UniqueName: \"kubernetes.io/projected/66794c35-2684-4c32-a7c4-7df95beb1b5c-kube-api-access-zskqf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5\" (UID: \"66794c35-2684-4c32-a7c4-7df95beb1b5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:00.631215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.630738 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e36a42-a6d4-449b-8923-f3a061fd58ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5\" (UID: \"80e36a42-a6d4-449b-8923-f3a061fd58ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:00.631215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.631166 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacb0496-adc4-4859-afe2-88a8c07d27b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj\" (UID: \"aacb0496-adc4-4859-afe2-88a8c07d27b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:00.637799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.631949 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66794c35-2684-4c32-a7c4-7df95beb1b5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5\" (UID: \"66794c35-2684-4c32-a7c4-7df95beb1b5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:00.637799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.632101 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxxxn\" (UniqueName: \"kubernetes.io/projected/52ff5a60-2de6-41b0-b7c9-0864c9086318-kube-api-access-hxxxn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6\" (UID: \"52ff5a60-2de6-41b0-b7c9-0864c9086318\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:00.637799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.632228 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858\" (UID: \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:00.637799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.632255 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhmk\" (UniqueName: \"kubernetes.io/projected/dd63cd40-3af1-49cb-b17b-5576a1f434cd-kube-api-access-tfhmk\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4\" (UID: \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:00.637799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.632490 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3eb218-be48-4f91-b4ca-94675bca35f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4\" (UID: \"bd3eb218-be48-4f91-b4ca-94675bca35f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:00.637799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.632558 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsx5p\" (UniqueName: \"kubernetes.io/projected/bd3eb218-be48-4f91-b4ca-94675bca35f5-kube-api-access-dsx5p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4\" (UID: \"bd3eb218-be48-4f91-b4ca-94675bca35f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:00.637799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.633168 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfd98\" (UniqueName: \"kubernetes.io/projected/80e36a42-a6d4-449b-8923-f3a061fd58ec-kube-api-access-kfd98\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5\" (UID: \"80e36a42-a6d4-449b-8923-f3a061fd58ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:00.664091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.655801 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:00.664091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.663813 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:00.685539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.685505 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnsdc\" (UniqueName: \"kubernetes.io/projected/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-kube-api-access-xnsdc\") pod \"maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858\" (UID: \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736154 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zskqf\" (UniqueName: \"kubernetes.io/projected/66794c35-2684-4c32-a7c4-7df95beb1b5c-kube-api-access-zskqf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5\" (UID: \"66794c35-2684-4c32-a7c4-7df95beb1b5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736259 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e36a42-a6d4-449b-8923-f3a061fd58ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5\" (UID: \"80e36a42-a6d4-449b-8923-f3a061fd58ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736338 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacb0496-adc4-4859-afe2-88a8c07d27b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj\" (UID: \"aacb0496-adc4-4859-afe2-88a8c07d27b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736464 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66794c35-2684-4c32-a7c4-7df95beb1b5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5\" (UID: \"66794c35-2684-4c32-a7c4-7df95beb1b5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736497 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxxxn\" (UniqueName: \"kubernetes.io/projected/52ff5a60-2de6-41b0-b7c9-0864c9086318-kube-api-access-hxxxn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6\" (UID: \"52ff5a60-2de6-41b0-b7c9-0864c9086318\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736543 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhmk\" (UniqueName: \"kubernetes.io/projected/dd63cd40-3af1-49cb-b17b-5576a1f434cd-kube-api-access-tfhmk\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4\" (UID: \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736574 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3eb218-be48-4f91-b4ca-94675bca35f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4\" (UID: \"bd3eb218-be48-4f91-b4ca-94675bca35f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736636 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsx5p\" (UniqueName: \"kubernetes.io/projected/bd3eb218-be48-4f91-b4ca-94675bca35f5-kube-api-access-dsx5p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4\" (UID: \"bd3eb218-be48-4f91-b4ca-94675bca35f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736784 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfd98\" (UniqueName: \"kubernetes.io/projected/80e36a42-a6d4-449b-8923-f3a061fd58ec-kube-api-access-kfd98\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5\" (UID: \"80e36a42-a6d4-449b-8923-f3a061fd58ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736875 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsftg\" (UniqueName: \"kubernetes.io/projected/aacb0496-adc4-4859-afe2-88a8c07d27b3-kube-api-access-rsftg\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj\" (UID: \"aacb0496-adc4-4859-afe2-88a8c07d27b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736923 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63cd40-3af1-49cb-b17b-5576a1f434cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4\" (UID: \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.736996 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ff5a60-2de6-41b0-b7c9-0864c9086318-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6\" (UID: \"52ff5a60-2de6-41b0-b7c9-0864c9086318\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:00.738398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.737812 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ff5a60-2de6-41b0-b7c9-0864c9086318-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6\" (UID: \"52ff5a60-2de6-41b0-b7c9-0864c9086318\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:00.742136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.740638 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3eb218-be48-4f91-b4ca-94675bca35f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4\" (UID: \"bd3eb218-be48-4f91-b4ca-94675bca35f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:00.742136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.741787 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66794c35-2684-4c32-a7c4-7df95beb1b5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5\" (UID: \"66794c35-2684-4c32-a7c4-7df95beb1b5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:00.743202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.743133 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63cd40-3af1-49cb-b17b-5576a1f434cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4\" (UID: \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:00.749681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.747222 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e36a42-a6d4-449b-8923-f3a061fd58ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5\" (UID: \"80e36a42-a6d4-449b-8923-f3a061fd58ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:00.749681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.747547 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacb0496-adc4-4859-afe2-88a8c07d27b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj\" (UID: \"aacb0496-adc4-4859-afe2-88a8c07d27b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:00.801836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.796270 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zskqf\" (UniqueName: \"kubernetes.io/projected/66794c35-2684-4c32-a7c4-7df95beb1b5c-kube-api-access-zskqf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5\" (UID: \"66794c35-2684-4c32-a7c4-7df95beb1b5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:00.810550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.810480 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsftg\" (UniqueName: \"kubernetes.io/projected/aacb0496-adc4-4859-afe2-88a8c07d27b3-kube-api-access-rsftg\") pod \"maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj\" (UID: \"aacb0496-adc4-4859-afe2-88a8c07d27b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:00.810948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.810887 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxxxn\" (UniqueName: \"kubernetes.io/projected/52ff5a60-2de6-41b0-b7c9-0864c9086318-kube-api-access-hxxxn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6\" (UID: \"52ff5a60-2de6-41b0-b7c9-0864c9086318\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:00.812027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.811971 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhmk\" (UniqueName: \"kubernetes.io/projected/dd63cd40-3af1-49cb-b17b-5576a1f434cd-kube-api-access-tfhmk\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4\" (UID: \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:00.813407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.813348 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfd98\" (UniqueName: \"kubernetes.io/projected/80e36a42-a6d4-449b-8923-f3a061fd58ec-kube-api-access-kfd98\") pod \"maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5\" (UID: \"80e36a42-a6d4-449b-8923-f3a061fd58ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:00.816308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.815387 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsx5p\" (UniqueName: \"kubernetes.io/projected/bd3eb218-be48-4f91-b4ca-94675bca35f5-kube-api-access-dsx5p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4\" (UID: \"bd3eb218-be48-4f91-b4ca-94675bca35f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:00.830307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.830066 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:00.842970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.842789 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:00.875690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.875664 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr"] Oct 29 23:45:00.885679 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:00.885630 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a37864a_0480_4fe9_b5c4_c946c1f049a9.slice/crio-121fca376f2e49eff18841cba9c03f05d46d6dd88f2da00d8c05e6fabe41c375 WatchSource:0}: Error finding container 121fca376f2e49eff18841cba9c03f05d46d6dd88f2da00d8c05e6fabe41c375: Status 404 returned error can't find the container with id 121fca376f2e49eff18841cba9c03f05d46d6dd88f2da00d8c05e6fabe41c375 Oct 29 23:45:00.911708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.907765 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:00.949270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.949193 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:00.960190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.960165 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:00.960610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.960398 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:00.979416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.972405 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:00.988803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.988609 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:00.994331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:00.994056 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:01.049007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.048971 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" event={"ID":"5a37864a-0480-4fe9-b5c4-c946c1f049a9","Type":"ContainerStarted","Data":"121fca376f2e49eff18841cba9c03f05d46d6dd88f2da00d8c05e6fabe41c375"} Oct 29 23:45:01.126035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.125956 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz"] Oct 29 23:45:01.138322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.138241 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct"] Oct 29 23:45:01.155363 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.154812 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr"] Oct 29 23:45:01.159701 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:01.156619 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1cc52d_92bf_4fa5_afb3_3e4b433f12aa.slice/crio-ec3c62eae3465baa38bcd3f46bfaca5a8da585dd67910886cad942cea65514b1 WatchSource:0}: Error finding container ec3c62eae3465baa38bcd3f46bfaca5a8da585dd67910886cad942cea65514b1: Status 404 returned error can't find the container with id ec3c62eae3465baa38bcd3f46bfaca5a8da585dd67910886cad942cea65514b1 Oct 29 23:45:01.351012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.350960 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth"] Oct 29 23:45:01.390164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.390084 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8"] Oct 29 23:45:01.403609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.397833 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4"] Oct 29 23:45:01.403813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.403794 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27"] Oct 29 23:45:01.453062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.453022 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg"] Oct 29 23:45:01.485850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.485814 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858"] Oct 29 23:45:01.534182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.534134 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5"] Oct 29 23:45:01.543858 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:01.543798 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde02fb5_67b1_4da5_9dc3_63bc2b930f08.slice/crio-872e5316c799b42b21ff3e0470b0110e62c519015d2aad5fb1dc503993e0d141 WatchSource:0}: Error finding container 872e5316c799b42b21ff3e0470b0110e62c519015d2aad5fb1dc503993e0d141: Status 404 returned error can't find the container with id 872e5316c799b42b21ff3e0470b0110e62c519015d2aad5fb1dc503993e0d141 Oct 29 23:45:01.560539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.560511 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5"] Oct 29 23:45:01.563898 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:01.563859 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e36a42_a6d4_449b_8923_f3a061fd58ec.slice/crio-23ef8f3573c97461cc8b3f9d3a501d9e9ce7681f9575d0c2797a14ca4c73ea87 WatchSource:0}: Error finding container 23ef8f3573c97461cc8b3f9d3a501d9e9ce7681f9575d0c2797a14ca4c73ea87: Status 404 returned error can't find the container with id 23ef8f3573c97461cc8b3f9d3a501d9e9ce7681f9575d0c2797a14ca4c73ea87 Oct 29 23:45:01.618665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.618574 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4"] Oct 29 23:45:01.627857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.626803 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk"] Oct 29 23:45:01.634889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.634170 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x"] Oct 29 23:45:01.775189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.775163 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4"] Oct 29 23:45:01.796320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.795397 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj"] Oct 29 23:45:01.801775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:01.801752 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6"] Oct 29 23:45:01.949148 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:01.949121 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3eb218_be48_4f91_b4ca_94675bca35f5.slice/crio-80d877966ed1b8d00d290d97906e7875a507ed9c2d1500771271e4ce9c0c2774 WatchSource:0}: Error finding container 80d877966ed1b8d00d290d97906e7875a507ed9c2d1500771271e4ce9c0c2774: Status 404 returned error can't find the container with id 80d877966ed1b8d00d290d97906e7875a507ed9c2d1500771271e4ce9c0c2774 Oct 29 23:45:01.955073 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:01.951253 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ff5a60_2de6_41b0_b7c9_0864c9086318.slice/crio-2e0c481ec16d4044d6ade7a9ec6377d8a2a8498cf82ebbe57e3efc6a6e989a4c WatchSource:0}: Error finding container 2e0c481ec16d4044d6ade7a9ec6377d8a2a8498cf82ebbe57e3efc6a6e989a4c: Status 404 returned error can't find the container with id 2e0c481ec16d4044d6ade7a9ec6377d8a2a8498cf82ebbe57e3efc6a6e989a4c Oct 29 23:45:02.063334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.063242 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" event={"ID":"5a37864a-0480-4fe9-b5c4-c946c1f049a9","Type":"ContainerStarted","Data":"e15f7afac309b018521e1acb693979ba518a27ff102d51c6a7b59161beb5f088"} Oct 29 23:45:02.064639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.064607 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" event={"ID":"4cb9077b-6b07-4a72-907e-b3e9b0e2534a","Type":"ContainerStarted","Data":"cd9477cc65910e0d8caee1629377d472b1cfe27b7751b5cda2cf06e0a418f801"} Oct 29 23:45:02.066005 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.065927 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" event={"ID":"8c9e79f9-8c22-4a9f-87db-bd6413409296","Type":"ContainerStarted","Data":"cc3d9953fe86bc572288aa9f8cd0042d3b2d5d0f94ea5ba264909b6640bd1e59"} Oct 29 23:45:02.067332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.067300 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" event={"ID":"bd3eb218-be48-4f91-b4ca-94675bca35f5","Type":"ContainerStarted","Data":"80d877966ed1b8d00d290d97906e7875a507ed9c2d1500771271e4ce9c0c2774"} Oct 29 23:45:02.078193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.078130 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" event={"ID":"dd63cd40-3af1-49cb-b17b-5576a1f434cd","Type":"ContainerStarted","Data":"656e37fd37d036fb9a343c58869926c47396b553d2d7dd43a50239ef11da9143"} Oct 29 23:45:02.083695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.083668 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" event={"ID":"aacb0496-adc4-4859-afe2-88a8c07d27b3","Type":"ContainerStarted","Data":"af4a4961da538bd7b1a2e926674d6a3ab4dc6c116c71bf5193cd2628ec0a9d35"} Oct 29 23:45:02.085239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.085212 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" event={"ID":"66794c35-2684-4c32-a7c4-7df95beb1b5c","Type":"ContainerStarted","Data":"24c0af018f0633b5981ee139012fcf25bf3860cb88411cd33ec612f46016c9d3"} Oct 29 23:45:02.086454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.086432 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" event={"ID":"137de700-2980-41fc-b778-24aa711d2ac2","Type":"ContainerStarted","Data":"7b007e27d3a0a36ec18781051c4ebc8d1ae9b8927df80942f83b2e4c2da3b655"} Oct 29 23:45:02.087774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.087748 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" event={"ID":"7055786b-a7d7-4894-81c3-65ac4348fea9","Type":"ContainerStarted","Data":"5ff398b0bc663f88362ad605c77ad14de564d70f1c47e86c750d3e929a067da2"} Oct 29 23:45:02.089110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.089086 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" event={"ID":"fefefc63-6416-4fec-b277-6a8b9e9de6fc","Type":"ContainerStarted","Data":"7abcbd61b058ddb5b5cb80a66d7b13971cd01841dfd982d62dfb8494a75e2e52"} Oct 29 23:45:02.090857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.090834 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" event={"ID":"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa","Type":"ContainerStarted","Data":"323c43fa71d101251a2ea1e4bd478f03f0ce4688c167b4a18cfba477b2de47fa"} Oct 29 23:45:02.090960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.090881 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" event={"ID":"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa","Type":"ContainerStarted","Data":"ec3c62eae3465baa38bcd3f46bfaca5a8da585dd67910886cad942cea65514b1"} Oct 29 23:45:02.092173 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.092150 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" event={"ID":"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb","Type":"ContainerStarted","Data":"f64de5aafa0c561351c2391af5441b3d6e8872df4a85936a87fdf42ab7bb0b67"} Oct 29 23:45:02.093420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.093397 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" event={"ID":"52ff5a60-2de6-41b0-b7c9-0864c9086318","Type":"ContainerStarted","Data":"2e0c481ec16d4044d6ade7a9ec6377d8a2a8498cf82ebbe57e3efc6a6e989a4c"} Oct 29 23:45:02.094628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.094604 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" event={"ID":"bde02fb5-67b1-4da5-9dc3-63bc2b930f08","Type":"ContainerStarted","Data":"872e5316c799b42b21ff3e0470b0110e62c519015d2aad5fb1dc503993e0d141"} Oct 29 23:45:02.095832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.095808 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" event={"ID":"80e36a42-a6d4-449b-8923-f3a061fd58ec","Type":"ContainerStarted","Data":"23ef8f3573c97461cc8b3f9d3a501d9e9ce7681f9575d0c2797a14ca4c73ea87"} Oct 29 23:45:02.097028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.097007 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" event={"ID":"1e5b2e09-ffc2-408c-a7a7-97b078c2984e","Type":"ContainerStarted","Data":"087f4f9169b6749c162f173f8764c2d95790e9003f2fc8a884f5950e87ab6c9e"} Oct 29 23:45:02.098216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.098190 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" event={"ID":"db376677-f21f-4807-ab75-b82ce5e8f51b","Type":"ContainerStarted","Data":"c8f3237a4c5bf2350165fa5aa1c74a03c5a92fafea366324bd19720fb422087d"} Oct 29 23:45:02.099671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.099641 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" event={"ID":"74c99f0d-50d0-4226-b5b7-fd1175bedb56","Type":"ContainerStarted","Data":"d79c01d3619d1a0025e8d0528d198d8247768a2e02b41fabe9e8b75c38ab6db8"} Oct 29 23:45:02.099765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.099680 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" event={"ID":"74c99f0d-50d0-4226-b5b7-fd1175bedb56","Type":"ContainerStarted","Data":"09ec3a273928be7137e0779b1031df4bf2db098a6d395c648b7eff6b301ed663"} Oct 29 23:45:02.109673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.109628 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" podStartSLOduration=2.109614699 podStartE2EDuration="2.109614699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:02.104909547 +0000 UTC m=+3699.220829153" watchObservedRunningTime="2026-10-29 23:45:02.109614699 +0000 UTC m=+3699.225534305" Oct 29 23:45:02.146534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.146481 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" podStartSLOduration=2.146465719 podStartE2EDuration="2.146465719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:02.145772673 +0000 UTC m=+3699.261692283" watchObservedRunningTime="2026-10-29 23:45:02.146465719 +0000 UTC m=+3699.262385329" Oct 29 23:45:02.187540 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:02.187477 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" podStartSLOduration=2.187463138 podStartE2EDuration="2.187463138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:02.183867638 +0000 UTC m=+3699.299787243" watchObservedRunningTime="2026-10-29 23:45:02.187463138 +0000 UTC m=+3699.303382744" Oct 29 23:45:03.114649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.114544 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" event={"ID":"fefefc63-6416-4fec-b277-6a8b9e9de6fc","Type":"ContainerStarted","Data":"1e66057564c295bd2ddbffda34a669e15886e35e9490557f667e32ebe4f00a1a"} Oct 29 23:45:03.116904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.116811 5736 generic.go:356] "Generic (PLEG): container finished" podID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerID="323c43fa71d101251a2ea1e4bd478f03f0ce4688c167b4a18cfba477b2de47fa" exitCode=60 Oct 29 23:45:03.117053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.116938 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" event={"ID":"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa","Type":"ContainerDied","Data":"323c43fa71d101251a2ea1e4bd478f03f0ce4688c167b4a18cfba477b2de47fa"} Oct 29 23:45:03.117354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.117335 5736 scope.go:122] "RemoveContainer" containerID="323c43fa71d101251a2ea1e4bd478f03f0ce4688c167b4a18cfba477b2de47fa" Oct 29 23:45:03.120310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.119937 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" event={"ID":"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb","Type":"ContainerStarted","Data":"92058f51033fe60c4799935e8db6780100b848b936e822d5b2800827013e617d"} Oct 29 23:45:03.123023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.122604 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" event={"ID":"52ff5a60-2de6-41b0-b7c9-0864c9086318","Type":"ContainerStarted","Data":"70b9b912f22f4749fd23cdfcaa72c2c3334c33030795b99f41654392d3fdadef"} Oct 29 23:45:03.125806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.125783 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" event={"ID":"bde02fb5-67b1-4da5-9dc3-63bc2b930f08","Type":"ContainerStarted","Data":"d46e389f6df5308d42f3cf6726088ef6956602ad115fc83416089eaaed4331a4"} Oct 29 23:45:03.127906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.127693 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" event={"ID":"80e36a42-a6d4-449b-8923-f3a061fd58ec","Type":"ContainerStarted","Data":"f91aee62e6fdff4aa920ff02221113ade0e313c2c4aead42909ffd695d729481"} Oct 29 23:45:03.129555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.129531 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" event={"ID":"1e5b2e09-ffc2-408c-a7a7-97b078c2984e","Type":"ContainerStarted","Data":"e005be50b08f3b6e5176cb09eb616ef5e83a6f6686818a041f4a30f322cd1982"} Oct 29 23:45:03.143962 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.143932 5736 generic.go:356] "Generic (PLEG): container finished" podID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerID="3ede89fa2ef638d2ba6b398f25944a94a60bee46d6bbf893ba9c18910580416c" exitCode=60 Oct 29 23:45:03.144105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.144036 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" event={"ID":"db376677-f21f-4807-ab75-b82ce5e8f51b","Type":"ContainerDied","Data":"3ede89fa2ef638d2ba6b398f25944a94a60bee46d6bbf893ba9c18910580416c"} Oct 29 23:45:03.144436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.144411 5736 scope.go:122] "RemoveContainer" containerID="3ede89fa2ef638d2ba6b398f25944a94a60bee46d6bbf893ba9c18910580416c" Oct 29 23:45:03.150144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.150101 5736 generic.go:356] "Generic (PLEG): container finished" podID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerID="d79c01d3619d1a0025e8d0528d198d8247768a2e02b41fabe9e8b75c38ab6db8" exitCode=60 Oct 29 23:45:03.150494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.150168 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" event={"ID":"74c99f0d-50d0-4226-b5b7-fd1175bedb56","Type":"ContainerDied","Data":"d79c01d3619d1a0025e8d0528d198d8247768a2e02b41fabe9e8b75c38ab6db8"} Oct 29 23:45:03.151519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.150517 5736 scope.go:122] "RemoveContainer" containerID="d79c01d3619d1a0025e8d0528d198d8247768a2e02b41fabe9e8b75c38ab6db8" Oct 29 23:45:03.162197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.162148 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" podStartSLOduration=3.1621322530000002 podStartE2EDuration="3.162132253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.155897984 +0000 UTC m=+3700.271817595" watchObservedRunningTime="2026-10-29 23:45:03.162132253 +0000 UTC m=+3700.278051860" Oct 29 23:45:03.169323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.169302 5736 generic.go:356] "Generic (PLEG): container finished" podID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerID="e15f7afac309b018521e1acb693979ba518a27ff102d51c6a7b59161beb5f088" exitCode=60 Oct 29 23:45:03.169441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.169387 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" event={"ID":"5a37864a-0480-4fe9-b5c4-c946c1f049a9","Type":"ContainerDied","Data":"e15f7afac309b018521e1acb693979ba518a27ff102d51c6a7b59161beb5f088"} Oct 29 23:45:03.169744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.169714 5736 scope.go:122] "RemoveContainer" containerID="e15f7afac309b018521e1acb693979ba518a27ff102d51c6a7b59161beb5f088" Oct 29 23:45:03.171387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.171363 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" event={"ID":"4cb9077b-6b07-4a72-907e-b3e9b0e2534a","Type":"ContainerStarted","Data":"54dedbf388d39c746de40177d170e577240a480d3ef1a176341a65e3c72a6ae8"} Oct 29 23:45:03.171775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.171753 5736 scope.go:122] "RemoveContainer" containerID="54dedbf388d39c746de40177d170e577240a480d3ef1a176341a65e3c72a6ae8" Oct 29 23:45:03.174087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.174059 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" event={"ID":"8c9e79f9-8c22-4a9f-87db-bd6413409296","Type":"ContainerStarted","Data":"506fd7c827b205357e6e1b912158f7750b4f17830cceacf509f23eccd81ace6b"} Oct 29 23:45:03.176527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.176500 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" event={"ID":"bd3eb218-be48-4f91-b4ca-94675bca35f5","Type":"ContainerStarted","Data":"7ab1fda15532c6b98ec4c4a66a5457b51658ad198b3e5309d6ea054841592cb6"} Oct 29 23:45:03.178539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.178514 5736 generic.go:356] "Generic (PLEG): container finished" podID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerID="19d63645908f885f34ed1eb4b2be3bae4f52a8c3105ac98c71e541bb4e95114a" exitCode=60 Oct 29 23:45:03.178649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.178626 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" event={"ID":"dd63cd40-3af1-49cb-b17b-5576a1f434cd","Type":"ContainerDied","Data":"19d63645908f885f34ed1eb4b2be3bae4f52a8c3105ac98c71e541bb4e95114a"} Oct 29 23:45:03.178918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.178902 5736 scope.go:122] "RemoveContainer" containerID="19d63645908f885f34ed1eb4b2be3bae4f52a8c3105ac98c71e541bb4e95114a" Oct 29 23:45:03.182787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.182763 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" event={"ID":"aacb0496-adc4-4859-afe2-88a8c07d27b3","Type":"ContainerStarted","Data":"8c896f47893817bc15ee18c9a43937d3f73a2809d0b89700e8a7a63c49904343"} Oct 29 23:45:03.186413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.186386 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" event={"ID":"66794c35-2684-4c32-a7c4-7df95beb1b5c","Type":"ContainerStarted","Data":"31c52900b24775b451359ac0f52b52640694afa9e4550c3ad5da81bd064a02b1"} Oct 29 23:45:03.188386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.188332 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" event={"ID":"137de700-2980-41fc-b778-24aa711d2ac2","Type":"ContainerStarted","Data":"43052cc2a38d843e388d997eb228d21464198ab3d33cf046b68d9d9616aac654"} Oct 29 23:45:03.189944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.189924 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" event={"ID":"7055786b-a7d7-4894-81c3-65ac4348fea9","Type":"ContainerStarted","Data":"809982cd45b1ce87d6453a0c8bb6e71a6d1fd3043c6100a29dc7b4fde731ea7e"} Oct 29 23:45:03.203369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.203315 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" podStartSLOduration=3.203299949 podStartE2EDuration="3.203299949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.197986989 +0000 UTC m=+3700.313906596" watchObservedRunningTime="2026-10-29 23:45:03.203299949 +0000 UTC m=+3700.319219560" Oct 29 23:45:03.487874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.487479 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" podStartSLOduration=3.487457396 podStartE2EDuration="3.487457396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.302399714 +0000 UTC m=+3700.418319321" watchObservedRunningTime="2026-10-29 23:45:03.487457396 +0000 UTC m=+3700.603376997" Oct 29 23:45:03.607181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.607121 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" podStartSLOduration=3.607102931 podStartE2EDuration="3.607102931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.484703009 +0000 UTC m=+3700.600622616" watchObservedRunningTime="2026-10-29 23:45:03.607102931 +0000 UTC m=+3700.723022538" Oct 29 23:45:03.683016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.664911 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" podStartSLOduration=3.664894399 podStartE2EDuration="3.664894399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.558307402 +0000 UTC m=+3700.674227020" watchObservedRunningTime="2026-10-29 23:45:03.664894399 +0000 UTC m=+3700.780814008" Oct 29 23:45:03.683016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.676994 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" podStartSLOduration=3.676976195 podStartE2EDuration="3.676976195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.660876795 +0000 UTC m=+3700.776796403" watchObservedRunningTime="2026-10-29 23:45:03.676976195 +0000 UTC m=+3700.792895803" Oct 29 23:45:03.734629 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:03.734575 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5b2e09_ffc2_408c_a7a7_97b078c2984e.slice/crio-conmon-e005be50b08f3b6e5176cb09eb616ef5e83a6f6686818a041f4a30f322cd1982.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5b2e09_ffc2_408c_a7a7_97b078c2984e.slice/crio-e005be50b08f3b6e5176cb09eb616ef5e83a6f6686818a041f4a30f322cd1982.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacb0496_adc4_4859_afe2_88a8c07d27b3.slice/crio-8c896f47893817bc15ee18c9a43937d3f73a2809d0b89700e8a7a63c49904343.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7055786b_a7d7_4894_81c3_65ac4348fea9.slice/crio-conmon-809982cd45b1ce87d6453a0c8bb6e71a6d1fd3043c6100a29dc7b4fde731ea7e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3eb218_be48_4f91_b4ca_94675bca35f5.slice/crio-conmon-7ab1fda15532c6b98ec4c4a66a5457b51658ad198b3e5309d6ea054841592cb6.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:45:03.751460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.751403 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" podStartSLOduration=3.751389536 podStartE2EDuration="3.751389536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.74654236 +0000 UTC m=+3700.862461965" watchObservedRunningTime="2026-10-29 23:45:03.751389536 +0000 UTC m=+3700.867309142" Oct 29 23:45:03.809242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.809139 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" podStartSLOduration=3.809124384 podStartE2EDuration="3.809124384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.803208262 +0000 UTC m=+3700.919127878" watchObservedRunningTime="2026-10-29 23:45:03.809124384 +0000 UTC m=+3700.925044025" Oct 29 23:45:03.885888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.885835 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" podStartSLOduration=3.885817307 podStartE2EDuration="3.885817307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.881404204 +0000 UTC m=+3700.997323810" watchObservedRunningTime="2026-10-29 23:45:03.885817307 +0000 UTC m=+3701.001736915" Oct 29 23:45:03.960304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:03.960196 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" podStartSLOduration=3.960177952 podStartE2EDuration="3.960177952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:03.959028259 +0000 UTC m=+3701.074947864" watchObservedRunningTime="2026-10-29 23:45:03.960177952 +0000 UTC m=+3701.076097560" Oct 29 23:45:04.002713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.002646 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" podStartSLOduration=4.002629144 podStartE2EDuration="4.002629144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:04.002147907 +0000 UTC m=+3701.118067512" watchObservedRunningTime="2026-10-29 23:45:04.002629144 +0000 UTC m=+3701.118548752" Oct 29 23:45:04.049718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.049662 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" podStartSLOduration=4.049646877 podStartE2EDuration="4.049646877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:04.045731854 +0000 UTC m=+3701.161651460" watchObservedRunningTime="2026-10-29 23:45:04.049646877 +0000 UTC m=+3701.165566483" Oct 29 23:45:04.203366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.203338 5736 generic.go:356] "Generic (PLEG): container finished" podID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerID="8c896f47893817bc15ee18c9a43937d3f73a2809d0b89700e8a7a63c49904343" exitCode=60 Oct 29 23:45:04.203776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.203425 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" event={"ID":"aacb0496-adc4-4859-afe2-88a8c07d27b3","Type":"ContainerDied","Data":"8c896f47893817bc15ee18c9a43937d3f73a2809d0b89700e8a7a63c49904343"} Oct 29 23:45:04.203776 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.203767 5736 scope.go:122] "RemoveContainer" containerID="8c896f47893817bc15ee18c9a43937d3f73a2809d0b89700e8a7a63c49904343" Oct 29 23:45:04.205393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.205370 5736 generic.go:356] "Generic (PLEG): container finished" podID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerID="31c52900b24775b451359ac0f52b52640694afa9e4550c3ad5da81bd064a02b1" exitCode=60 Oct 29 23:45:04.205510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.205451 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" event={"ID":"66794c35-2684-4c32-a7c4-7df95beb1b5c","Type":"ContainerDied","Data":"31c52900b24775b451359ac0f52b52640694afa9e4550c3ad5da81bd064a02b1"} Oct 29 23:45:04.205799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.205776 5736 scope.go:122] "RemoveContainer" containerID="31c52900b24775b451359ac0f52b52640694afa9e4550c3ad5da81bd064a02b1" Oct 29 23:45:04.215862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.215836 5736 generic.go:356] "Generic (PLEG): container finished" podID="137de700-2980-41fc-b778-24aa711d2ac2" containerID="43052cc2a38d843e388d997eb228d21464198ab3d33cf046b68d9d9616aac654" exitCode=60 Oct 29 23:45:04.215945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.215917 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" event={"ID":"137de700-2980-41fc-b778-24aa711d2ac2","Type":"ContainerDied","Data":"43052cc2a38d843e388d997eb228d21464198ab3d33cf046b68d9d9616aac654"} Oct 29 23:45:04.216307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.216218 5736 scope.go:122] "RemoveContainer" containerID="43052cc2a38d843e388d997eb228d21464198ab3d33cf046b68d9d9616aac654" Oct 29 23:45:04.217786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.217762 5736 generic.go:356] "Generic (PLEG): container finished" podID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerID="809982cd45b1ce87d6453a0c8bb6e71a6d1fd3043c6100a29dc7b4fde731ea7e" exitCode=60 Oct 29 23:45:04.217875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.217844 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" event={"ID":"7055786b-a7d7-4894-81c3-65ac4348fea9","Type":"ContainerDied","Data":"809982cd45b1ce87d6453a0c8bb6e71a6d1fd3043c6100a29dc7b4fde731ea7e"} Oct 29 23:45:04.218064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.218047 5736 scope.go:122] "RemoveContainer" containerID="809982cd45b1ce87d6453a0c8bb6e71a6d1fd3043c6100a29dc7b4fde731ea7e" Oct 29 23:45:04.219887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.219854 5736 generic.go:356] "Generic (PLEG): container finished" podID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerID="1e66057564c295bd2ddbffda34a669e15886e35e9490557f667e32ebe4f00a1a" exitCode=60 Oct 29 23:45:04.220210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.220187 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" event={"ID":"fefefc63-6416-4fec-b277-6a8b9e9de6fc","Type":"ContainerDied","Data":"1e66057564c295bd2ddbffda34a669e15886e35e9490557f667e32ebe4f00a1a"} Oct 29 23:45:04.220380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.220358 5736 scope.go:122] "RemoveContainer" containerID="1e66057564c295bd2ddbffda34a669e15886e35e9490557f667e32ebe4f00a1a" Oct 29 23:45:04.234352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.225549 5736 generic.go:356] "Generic (PLEG): container finished" podID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerID="92058f51033fe60c4799935e8db6780100b848b936e822d5b2800827013e617d" exitCode=60 Oct 29 23:45:04.234352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.225752 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" event={"ID":"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb","Type":"ContainerDied","Data":"92058f51033fe60c4799935e8db6780100b848b936e822d5b2800827013e617d"} Oct 29 23:45:04.234352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.226217 5736 scope.go:122] "RemoveContainer" containerID="92058f51033fe60c4799935e8db6780100b848b936e822d5b2800827013e617d" Oct 29 23:45:04.238171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.238101 5736 generic.go:356] "Generic (PLEG): container finished" podID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerID="70b9b912f22f4749fd23cdfcaa72c2c3334c33030795b99f41654392d3fdadef" exitCode=60 Oct 29 23:45:04.238894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.238869 5736 scope.go:122] "RemoveContainer" containerID="70b9b912f22f4749fd23cdfcaa72c2c3334c33030795b99f41654392d3fdadef" Oct 29 23:45:04.239559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.239529 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" event={"ID":"52ff5a60-2de6-41b0-b7c9-0864c9086318","Type":"ContainerDied","Data":"70b9b912f22f4749fd23cdfcaa72c2c3334c33030795b99f41654392d3fdadef"} Oct 29 23:45:04.243333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.243306 5736 generic.go:356] "Generic (PLEG): container finished" podID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerID="d46e389f6df5308d42f3cf6726088ef6956602ad115fc83416089eaaed4331a4" exitCode=60 Oct 29 23:45:04.243618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.243568 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" event={"ID":"bde02fb5-67b1-4da5-9dc3-63bc2b930f08","Type":"ContainerDied","Data":"d46e389f6df5308d42f3cf6726088ef6956602ad115fc83416089eaaed4331a4"} Oct 29 23:45:04.244167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.244144 5736 scope.go:122] "RemoveContainer" containerID="d46e389f6df5308d42f3cf6726088ef6956602ad115fc83416089eaaed4331a4" Oct 29 23:45:04.247483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.247457 5736 generic.go:356] "Generic (PLEG): container finished" podID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerID="f91aee62e6fdff4aa920ff02221113ade0e313c2c4aead42909ffd695d729481" exitCode=60 Oct 29 23:45:04.247574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.247542 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" event={"ID":"80e36a42-a6d4-449b-8923-f3a061fd58ec","Type":"ContainerDied","Data":"f91aee62e6fdff4aa920ff02221113ade0e313c2c4aead42909ffd695d729481"} Oct 29 23:45:04.247958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.247936 5736 scope.go:122] "RemoveContainer" containerID="f91aee62e6fdff4aa920ff02221113ade0e313c2c4aead42909ffd695d729481" Oct 29 23:45:04.250120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.250089 5736 generic.go:356] "Generic (PLEG): container finished" podID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerID="e005be50b08f3b6e5176cb09eb616ef5e83a6f6686818a041f4a30f322cd1982" exitCode=60 Oct 29 23:45:04.250403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.250377 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" event={"ID":"1e5b2e09-ffc2-408c-a7a7-97b078c2984e","Type":"ContainerDied","Data":"e005be50b08f3b6e5176cb09eb616ef5e83a6f6686818a041f4a30f322cd1982"} Oct 29 23:45:04.250846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.250804 5736 scope.go:122] "RemoveContainer" containerID="e005be50b08f3b6e5176cb09eb616ef5e83a6f6686818a041f4a30f322cd1982" Oct 29 23:45:04.253988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.253968 5736 generic.go:356] "Generic (PLEG): container finished" podID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerID="54dedbf388d39c746de40177d170e577240a480d3ef1a176341a65e3c72a6ae8" exitCode=60 Oct 29 23:45:04.254078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.254039 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" event={"ID":"4cb9077b-6b07-4a72-907e-b3e9b0e2534a","Type":"ContainerDied","Data":"54dedbf388d39c746de40177d170e577240a480d3ef1a176341a65e3c72a6ae8"} Oct 29 23:45:04.255734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.255711 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" event={"ID":"8c9e79f9-8c22-4a9f-87db-bd6413409296","Type":"ContainerDied","Data":"506fd7c827b205357e6e1b912158f7750b4f17830cceacf509f23eccd81ace6b"} Oct 29 23:45:04.255890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.255697 5736 generic.go:356] "Generic (PLEG): container finished" podID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerID="506fd7c827b205357e6e1b912158f7750b4f17830cceacf509f23eccd81ace6b" exitCode=60 Oct 29 23:45:04.256269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.256223 5736 scope.go:122] "RemoveContainer" containerID="506fd7c827b205357e6e1b912158f7750b4f17830cceacf509f23eccd81ace6b" Oct 29 23:45:04.258456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.258433 5736 generic.go:356] "Generic (PLEG): container finished" podID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerID="7ab1fda15532c6b98ec4c4a66a5457b51658ad198b3e5309d6ea054841592cb6" exitCode=60 Oct 29 23:45:04.258539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.258462 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" event={"ID":"bd3eb218-be48-4f91-b4ca-94675bca35f5","Type":"ContainerDied","Data":"7ab1fda15532c6b98ec4c4a66a5457b51658ad198b3e5309d6ea054841592cb6"} Oct 29 23:45:04.258803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:04.258784 5736 scope.go:122] "RemoveContainer" containerID="7ab1fda15532c6b98ec4c4a66a5457b51658ad198b3e5309d6ea054841592cb6" Oct 29 23:45:05.516158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:05.516116 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" event={"ID":"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa","Type":"ContainerStarted","Data":"b394b888f76457d24a4ee6c9b8255291a2cc15c8a01335e33bd25412116e4d4f"} Oct 29 23:45:05.533200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:05.533140 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" event={"ID":"db376677-f21f-4807-ab75-b82ce5e8f51b","Type":"ContainerStarted","Data":"3f2bf5fb96debce9e2b80f27232812051f16a422ffd2b10df1a18478f0770df5"} Oct 29 23:45:05.582156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:05.573263 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" event={"ID":"4cb9077b-6b07-4a72-907e-b3e9b0e2534a","Type":"ContainerStarted","Data":"1e8260565a096ef3eb110e7ac2efaaea977716d63373ade0b896fcbf613040dd"} Oct 29 23:45:05.661698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:05.649732 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" podStartSLOduration=5.649711998 podStartE2EDuration="5.649711998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:05.646226436 +0000 UTC m=+3702.762146046" watchObservedRunningTime="2026-10-29 23:45:05.649711998 +0000 UTC m=+3702.765631618" Oct 29 23:45:05.695655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:05.695426 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" podStartSLOduration=5.69540644 podStartE2EDuration="5.69540644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:05.69238629 +0000 UTC m=+3702.808305897" watchObservedRunningTime="2026-10-29 23:45:05.69540644 +0000 UTC m=+3702.811326046" Oct 29 23:45:06.738668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.738616 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j"] Oct 29 23:45:06.847901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.847862 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" event={"ID":"bde02fb5-67b1-4da5-9dc3-63bc2b930f08","Type":"ContainerStarted","Data":"babedd511fe15a356eedd3768cbc91f5b8c607b0b05dc13be91909205395985b"} Oct 29 23:45:06.848699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.848675 5736 scope.go:122] "RemoveContainer" containerID="babedd511fe15a356eedd3768cbc91f5b8c607b0b05dc13be91909205395985b" Oct 29 23:45:06.849286 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.849254 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858_odh-ai-gateway-infra(bde02fb5-67b1-4da5-9dc3-63bc2b930f08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" Oct 29 23:45:06.881828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.881746 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" event={"ID":"80e36a42-a6d4-449b-8923-f3a061fd58ec","Type":"ContainerStarted","Data":"ce8d06d7ba0380cd16eff3b47ddb5060671b5ded3aa18d4f96c64a6c1de37acb"} Oct 29 23:45:06.904604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.893844 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" event={"ID":"1e5b2e09-ffc2-408c-a7a7-97b078c2984e","Type":"ContainerStarted","Data":"e9e2dcc7d16d32e53218391658927434472eaf418e60e702b4d08d3dc2cd16d2"} Oct 29 23:45:06.904604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.894388 5736 scope.go:122] "RemoveContainer" containerID="e9e2dcc7d16d32e53218391658927434472eaf418e60e702b4d08d3dc2cd16d2" Oct 29 23:45:06.904604 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.895896 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x_odh-ai-gateway-infra(1e5b2e09-ffc2-408c-a7a7-97b078c2984e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" Oct 29 23:45:06.904604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.901474 5736 generic.go:356] "Generic (PLEG): container finished" podID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerID="3f2bf5fb96debce9e2b80f27232812051f16a422ffd2b10df1a18478f0770df5" exitCode=60 Oct 29 23:45:06.904604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.901558 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" event={"ID":"db376677-f21f-4807-ab75-b82ce5e8f51b","Type":"ContainerDied","Data":"3f2bf5fb96debce9e2b80f27232812051f16a422ffd2b10df1a18478f0770df5"} Oct 29 23:45:06.904604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.901705 5736 scope.go:122] "RemoveContainer" containerID="3ede89fa2ef638d2ba6b398f25944a94a60bee46d6bbf893ba9c18910580416c" Oct 29 23:45:06.904604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.902254 5736 scope.go:122] "RemoveContainer" containerID="3f2bf5fb96debce9e2b80f27232812051f16a422ffd2b10df1a18478f0770df5" Oct 29 23:45:06.904604 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.902546 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth_odh-ai-gateway-infra(db376677-f21f-4807-ab75-b82ce5e8f51b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" Oct 29 23:45:06.905204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.904973 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" event={"ID":"74c99f0d-50d0-4226-b5b7-fd1175bedb56","Type":"ContainerStarted","Data":"51c0e40b09398c2c99e4b1f1b64e2dd1abad50b5dcf6e53c5a9f4637ea735b59"} Oct 29 23:45:06.908119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.905848 5736 scope.go:122] "RemoveContainer" containerID="51c0e40b09398c2c99e4b1f1b64e2dd1abad50b5dcf6e53c5a9f4637ea735b59" Oct 29 23:45:06.908119 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.906106 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz_odh-ai-gateway-infra(74c99f0d-50d0-4226-b5b7-fd1175bedb56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" Oct 29 23:45:06.908119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.907299 5736 generic.go:356] "Generic (PLEG): container finished" podID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerID="dfc93df4c6cb6dd2c7d0594f4e431604fc88c7477bc661d82d01da8ed6886f52" exitCode=60 Oct 29 23:45:06.908119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.907388 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" event={"ID":"5a37864a-0480-4fe9-b5c4-c946c1f049a9","Type":"ContainerDied","Data":"dfc93df4c6cb6dd2c7d0594f4e431604fc88c7477bc661d82d01da8ed6886f52"} Oct 29 23:45:06.908119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.907868 5736 scope.go:122] "RemoveContainer" containerID="dfc93df4c6cb6dd2c7d0594f4e431604fc88c7477bc661d82d01da8ed6886f52" Oct 29 23:45:06.908475 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.908234 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr_odh-ai-gateway-infra(5a37864a-0480-4fe9-b5c4-c946c1f049a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" Oct 29 23:45:06.911561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.910425 5736 generic.go:356] "Generic (PLEG): container finished" podID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerID="1e8260565a096ef3eb110e7ac2efaaea977716d63373ade0b896fcbf613040dd" exitCode=60 Oct 29 23:45:06.911561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.910482 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" event={"ID":"4cb9077b-6b07-4a72-907e-b3e9b0e2534a","Type":"ContainerDied","Data":"1e8260565a096ef3eb110e7ac2efaaea977716d63373ade0b896fcbf613040dd"} Oct 29 23:45:06.911561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.910811 5736 scope.go:122] "RemoveContainer" containerID="1e8260565a096ef3eb110e7ac2efaaea977716d63373ade0b896fcbf613040dd" Oct 29 23:45:06.911561 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.911138 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8_odh-ai-gateway-infra(4cb9077b-6b07-4a72-907e-b3e9b0e2534a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" Oct 29 23:45:06.913737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.913712 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" event={"ID":"8c9e79f9-8c22-4a9f-87db-bd6413409296","Type":"ContainerStarted","Data":"79b67cf8f1b06e0375da141a7d4d84cba9648bb3e8a4de633404fa82a58bf587"} Oct 29 23:45:06.914055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.914035 5736 scope.go:122] "RemoveContainer" containerID="79b67cf8f1b06e0375da141a7d4d84cba9648bb3e8a4de633404fa82a58bf587" Oct 29 23:45:06.914319 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.914294 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4_odh-ai-gateway-infra(8c9e79f9-8c22-4a9f-87db-bd6413409296)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" Oct 29 23:45:06.916969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.916628 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" event={"ID":"bd3eb218-be48-4f91-b4ca-94675bca35f5","Type":"ContainerStarted","Data":"124019b61304223e01bd7a0ac6f8824ea3911111cb02f99dc37a5bf6285e0f4e"} Oct 29 23:45:06.916969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.916715 5736 scope.go:122] "RemoveContainer" containerID="124019b61304223e01bd7a0ac6f8824ea3911111cb02f99dc37a5bf6285e0f4e" Oct 29 23:45:06.916969 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.916932 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4_odh-ai-gateway-infra(bd3eb218-be48-4f91-b4ca-94675bca35f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" Oct 29 23:45:06.919727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.919159 5736 generic.go:356] "Generic (PLEG): container finished" podID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerID="7623efecc41e6bc1b586cdfe6496550c6d0e64a6dd5f747e6bb2760c7cadcc4f" exitCode=60 Oct 29 23:45:06.919727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.919233 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" event={"ID":"dd63cd40-3af1-49cb-b17b-5576a1f434cd","Type":"ContainerDied","Data":"7623efecc41e6bc1b586cdfe6496550c6d0e64a6dd5f747e6bb2760c7cadcc4f"} Oct 29 23:45:06.919727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.919458 5736 scope.go:122] "RemoveContainer" containerID="7623efecc41e6bc1b586cdfe6496550c6d0e64a6dd5f747e6bb2760c7cadcc4f" Oct 29 23:45:06.919727 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.919692 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4_odh-ai-gateway-infra(dd63cd40-3af1-49cb-b17b-5576a1f434cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" Oct 29 23:45:06.921075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.921061 5736 scope.go:122] "RemoveContainer" containerID="e15f7afac309b018521e1acb693979ba518a27ff102d51c6a7b59161beb5f088" Oct 29 23:45:06.923408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.922492 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" event={"ID":"aacb0496-adc4-4859-afe2-88a8c07d27b3","Type":"ContainerStarted","Data":"7ebc6519b6da517c5cad02e2aa463672195583bbc742d1e233685d390240411e"} Oct 29 23:45:06.923408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.923186 5736 scope.go:122] "RemoveContainer" containerID="7ebc6519b6da517c5cad02e2aa463672195583bbc742d1e233685d390240411e" Oct 29 23:45:06.923545 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.923425 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj_odh-ai-gateway-infra(aacb0496-adc4-4859-afe2-88a8c07d27b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" Oct 29 23:45:06.924943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.924919 5736 generic.go:356] "Generic (PLEG): container finished" podID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerID="f92f51093d4bc19624ae5a30f27814b2a4a0ae64f2d32890c50baa8b942525e9" exitCode=60 Oct 29 23:45:06.925016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.924980 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" event={"ID":"66794c35-2684-4c32-a7c4-7df95beb1b5c","Type":"ContainerDied","Data":"f92f51093d4bc19624ae5a30f27814b2a4a0ae64f2d32890c50baa8b942525e9"} Oct 29 23:45:06.925275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.925260 5736 scope.go:122] "RemoveContainer" containerID="f92f51093d4bc19624ae5a30f27814b2a4a0ae64f2d32890c50baa8b942525e9" Oct 29 23:45:06.925505 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.925487 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5_odh-ai-gateway-infra(66794c35-2684-4c32-a7c4-7df95beb1b5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" Oct 29 23:45:06.928130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.928105 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" event={"ID":"137de700-2980-41fc-b778-24aa711d2ac2","Type":"ContainerStarted","Data":"de776c564ffa0701f5cfbecd2e3cfefa24d39c8125cd83bc0952e5746a1ab4b5"} Oct 29 23:45:06.928441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.928422 5736 scope.go:122] "RemoveContainer" containerID="de776c564ffa0701f5cfbecd2e3cfefa24d39c8125cd83bc0952e5746a1ab4b5" Oct 29 23:45:06.928728 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.928708 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr_odh-ai-gateway-infra(137de700-2980-41fc-b778-24aa711d2ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" podUID="137de700-2980-41fc-b778-24aa711d2ac2" Oct 29 23:45:06.931319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.930821 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" event={"ID":"7055786b-a7d7-4894-81c3-65ac4348fea9","Type":"ContainerStarted","Data":"03d5d714d8fab3c0c20a154384fcb006372bbedf7fc8fd50c252cc1585ddae0b"} Oct 29 23:45:06.931319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.931020 5736 scope.go:122] "RemoveContainer" containerID="03d5d714d8fab3c0c20a154384fcb006372bbedf7fc8fd50c252cc1585ddae0b" Oct 29 23:45:06.931319 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.931266 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg_odh-ai-gateway-infra(7055786b-a7d7-4894-81c3-65ac4348fea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" Oct 29 23:45:06.940685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.940435 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h"] Oct 29 23:45:06.943045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.943016 5736 scope.go:122] "RemoveContainer" containerID="991309b8643dd62e1ac9ec462059c6b50731b0a9142c1feede87d9b53abd47fe" Oct 29 23:45:06.943423 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.943396 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27_odh-ai-gateway-infra(fefefc63-6416-4fec-b277-6a8b9e9de6fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" Oct 29 23:45:06.945320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.945292 5736 generic.go:356] "Generic (PLEG): container finished" podID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerID="b394b888f76457d24a4ee6c9b8255291a2cc15c8a01335e33bd25412116e4d4f" exitCode=60 Oct 29 23:45:06.945766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.945741 5736 scope.go:122] "RemoveContainer" containerID="b394b888f76457d24a4ee6c9b8255291a2cc15c8a01335e33bd25412116e4d4f" Oct 29 23:45:06.946165 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.946138 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct_odh-ai-gateway-infra(bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" Oct 29 23:45:06.948087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.948064 5736 scope.go:122] "RemoveContainer" containerID="dc82987dea9e983ac77edc31f38398e51b5a4a17fa607451b9544de9de0479f8" Oct 29 23:45:06.948219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.948189 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" event={"ID":"fefefc63-6416-4fec-b277-6a8b9e9de6fc","Type":"ContainerStarted","Data":"991309b8643dd62e1ac9ec462059c6b50731b0a9142c1feede87d9b53abd47fe"} Oct 29 23:45:06.948310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.948235 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" event={"ID":"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa","Type":"ContainerDied","Data":"b394b888f76457d24a4ee6c9b8255291a2cc15c8a01335e33bd25412116e4d4f"} Oct 29 23:45:06.948310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.948262 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" event={"ID":"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb","Type":"ContainerStarted","Data":"dc82987dea9e983ac77edc31f38398e51b5a4a17fa607451b9544de9de0479f8"} Oct 29 23:45:06.948473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.948315 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:06.948473 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:06.948342 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk_odh-ai-gateway-infra(b61d2aa1-55ea-46a6-adbd-f7a03cf883fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" Oct 29 23:45:06.958954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.958870 5736 scope.go:122] "RemoveContainer" containerID="54dedbf388d39c746de40177d170e577240a480d3ef1a176341a65e3c72a6ae8" Oct 29 23:45:06.964193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.964159 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:45:06.971193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.971163 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h"] Oct 29 23:45:06.999827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:06.999799 5736 scope.go:122] "RemoveContainer" containerID="19d63645908f885f34ed1eb4b2be3bae4f52a8c3105ac98c71e541bb4e95114a" Oct 29 23:45:07.022222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.022195 5736 scope.go:122] "RemoveContainer" containerID="31c52900b24775b451359ac0f52b52640694afa9e4550c3ad5da81bd064a02b1" Oct 29 23:45:07.039412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.039384 5736 scope.go:122] "RemoveContainer" containerID="323c43fa71d101251a2ea1e4bd478f03f0ce4688c167b4a18cfba477b2de47fa" Oct 29 23:45:07.105743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.104558 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.105743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.104966 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvwcb\" (UniqueName: \"kubernetes.io/projected/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-kube-api-access-kvwcb\") pod \"maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.105743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.105117 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.207147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.207049 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.207353 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.207294 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.207533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.207514 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvwcb\" (UniqueName: \"kubernetes.io/projected/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-kube-api-access-kvwcb\") pod \"maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.208040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.208006 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.210095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.210066 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.315603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.305158 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvwcb\" (UniqueName: \"kubernetes.io/projected/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-kube-api-access-kvwcb\") pod \"maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.571163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.571130 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:07.894608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.894555 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h"] Oct 29 23:45:07.894978 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:07.894554 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b94d6d5_7d12_4c1e_9a2d_060b12d6e338.slice/crio-ee867950028eee512f867f4bddea2eedb6c2b12491d9a033e1f26150b6712b4b WatchSource:0}: Error finding container ee867950028eee512f867f4bddea2eedb6c2b12491d9a033e1f26150b6712b4b: Status 404 returned error can't find the container with id ee867950028eee512f867f4bddea2eedb6c2b12491d9a033e1f26150b6712b4b Oct 29 23:45:07.974554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.974524 5736 scope.go:122] "RemoveContainer" containerID="7623efecc41e6bc1b586cdfe6496550c6d0e64a6dd5f747e6bb2760c7cadcc4f" Oct 29 23:45:07.975012 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.974973 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4_odh-ai-gateway-infra(dd63cd40-3af1-49cb-b17b-5576a1f434cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" Oct 29 23:45:07.976374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.976338 5736 generic.go:356] "Generic (PLEG): container finished" podID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerID="7ebc6519b6da517c5cad02e2aa463672195583bbc742d1e233685d390240411e" exitCode=60 Oct 29 23:45:07.976477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.976416 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" event={"ID":"aacb0496-adc4-4859-afe2-88a8c07d27b3","Type":"ContainerDied","Data":"7ebc6519b6da517c5cad02e2aa463672195583bbc742d1e233685d390240411e"} Oct 29 23:45:07.976477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.976471 5736 scope.go:122] "RemoveContainer" containerID="8c896f47893817bc15ee18c9a43937d3f73a2809d0b89700e8a7a63c49904343" Oct 29 23:45:07.976885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.976865 5736 scope.go:122] "RemoveContainer" containerID="7ebc6519b6da517c5cad02e2aa463672195583bbc742d1e233685d390240411e" Oct 29 23:45:07.977295 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.977268 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj_odh-ai-gateway-infra(aacb0496-adc4-4859-afe2-88a8c07d27b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" Oct 29 23:45:07.978638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.978619 5736 scope.go:122] "RemoveContainer" containerID="f92f51093d4bc19624ae5a30f27814b2a4a0ae64f2d32890c50baa8b942525e9" Oct 29 23:45:07.978945 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.978910 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5_odh-ai-gateway-infra(66794c35-2684-4c32-a7c4-7df95beb1b5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" Oct 29 23:45:07.980777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.980758 5736 generic.go:356] "Generic (PLEG): container finished" podID="137de700-2980-41fc-b778-24aa711d2ac2" containerID="de776c564ffa0701f5cfbecd2e3cfefa24d39c8125cd83bc0952e5746a1ab4b5" exitCode=60 Oct 29 23:45:07.980920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.980794 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" event={"ID":"137de700-2980-41fc-b778-24aa711d2ac2","Type":"ContainerDied","Data":"de776c564ffa0701f5cfbecd2e3cfefa24d39c8125cd83bc0952e5746a1ab4b5"} Oct 29 23:45:07.981096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.981075 5736 scope.go:122] "RemoveContainer" containerID="de776c564ffa0701f5cfbecd2e3cfefa24d39c8125cd83bc0952e5746a1ab4b5" Oct 29 23:45:07.981386 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.981362 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr_odh-ai-gateway-infra(137de700-2980-41fc-b778-24aa711d2ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" podUID="137de700-2980-41fc-b778-24aa711d2ac2" Oct 29 23:45:07.983148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.983128 5736 generic.go:356] "Generic (PLEG): container finished" podID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerID="03d5d714d8fab3c0c20a154384fcb006372bbedf7fc8fd50c252cc1585ddae0b" exitCode=60 Oct 29 23:45:07.983302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.983203 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" event={"ID":"7055786b-a7d7-4894-81c3-65ac4348fea9","Type":"ContainerDied","Data":"03d5d714d8fab3c0c20a154384fcb006372bbedf7fc8fd50c252cc1585ddae0b"} Oct 29 23:45:07.983516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.983496 5736 scope.go:122] "RemoveContainer" containerID="03d5d714d8fab3c0c20a154384fcb006372bbedf7fc8fd50c252cc1585ddae0b" Oct 29 23:45:07.983833 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.983810 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg_odh-ai-gateway-infra(7055786b-a7d7-4894-81c3-65ac4348fea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" Oct 29 23:45:07.985663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.985639 5736 generic.go:356] "Generic (PLEG): container finished" podID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerID="991309b8643dd62e1ac9ec462059c6b50731b0a9142c1feede87d9b53abd47fe" exitCode=60 Oct 29 23:45:07.985757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.985718 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" event={"ID":"fefefc63-6416-4fec-b277-6a8b9e9de6fc","Type":"ContainerDied","Data":"991309b8643dd62e1ac9ec462059c6b50731b0a9142c1feede87d9b53abd47fe"} Oct 29 23:45:07.985971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.985953 5736 scope.go:122] "RemoveContainer" containerID="991309b8643dd62e1ac9ec462059c6b50731b0a9142c1feede87d9b53abd47fe" Oct 29 23:45:07.986239 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.986213 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27_odh-ai-gateway-infra(fefefc63-6416-4fec-b277-6a8b9e9de6fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" Oct 29 23:45:07.987867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.987848 5736 scope.go:122] "RemoveContainer" containerID="b394b888f76457d24a4ee6c9b8255291a2cc15c8a01335e33bd25412116e4d4f" Oct 29 23:45:07.988135 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.988099 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct_odh-ai-gateway-infra(bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" Oct 29 23:45:07.989193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.989159 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" event={"ID":"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338","Type":"ContainerStarted","Data":"ee867950028eee512f867f4bddea2eedb6c2b12491d9a033e1f26150b6712b4b"} Oct 29 23:45:07.991354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.991333 5736 generic.go:356] "Generic (PLEG): container finished" podID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerID="dc82987dea9e983ac77edc31f38398e51b5a4a17fa607451b9544de9de0479f8" exitCode=60 Oct 29 23:45:07.991652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.991628 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" event={"ID":"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb","Type":"ContainerDied","Data":"dc82987dea9e983ac77edc31f38398e51b5a4a17fa607451b9544de9de0479f8"} Oct 29 23:45:07.991743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.991659 5736 scope.go:122] "RemoveContainer" containerID="dc82987dea9e983ac77edc31f38398e51b5a4a17fa607451b9544de9de0479f8" Oct 29 23:45:07.991943 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.991907 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk_odh-ai-gateway-infra(b61d2aa1-55ea-46a6-adbd-f7a03cf883fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" Oct 29 23:45:07.994020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.993997 5736 generic.go:356] "Generic (PLEG): container finished" podID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerID="5e24234f0dd68d39474472677ce16deb35e4d6e23b1fd9cf6855c429d4e67eed" exitCode=60 Oct 29 23:45:07.994133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.994073 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" event={"ID":"52ff5a60-2de6-41b0-b7c9-0864c9086318","Type":"ContainerDied","Data":"5e24234f0dd68d39474472677ce16deb35e4d6e23b1fd9cf6855c429d4e67eed"} Oct 29 23:45:07.994411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.994394 5736 scope.go:122] "RemoveContainer" containerID="5e24234f0dd68d39474472677ce16deb35e4d6e23b1fd9cf6855c429d4e67eed" Oct 29 23:45:07.994719 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.994692 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6_odh-ai-gateway-infra(52ff5a60-2de6-41b0-b7c9-0864c9086318)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" Oct 29 23:45:07.996128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.996103 5736 scope.go:122] "RemoveContainer" containerID="43052cc2a38d843e388d997eb228d21464198ab3d33cf046b68d9d9616aac654" Oct 29 23:45:07.996239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.996218 5736 generic.go:356] "Generic (PLEG): container finished" podID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerID="babedd511fe15a356eedd3768cbc91f5b8c607b0b05dc13be91909205395985b" exitCode=60 Oct 29 23:45:07.996317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.996298 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" event={"ID":"bde02fb5-67b1-4da5-9dc3-63bc2b930f08","Type":"ContainerDied","Data":"babedd511fe15a356eedd3768cbc91f5b8c607b0b05dc13be91909205395985b"} Oct 29 23:45:07.996556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.996536 5736 scope.go:122] "RemoveContainer" containerID="babedd511fe15a356eedd3768cbc91f5b8c607b0b05dc13be91909205395985b" Oct 29 23:45:07.996831 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.996809 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858_odh-ai-gateway-infra(bde02fb5-67b1-4da5-9dc3-63bc2b930f08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" Oct 29 23:45:07.998261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.998242 5736 generic.go:356] "Generic (PLEG): container finished" podID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerID="ce8d06d7ba0380cd16eff3b47ddb5060671b5ded3aa18d4f96c64a6c1de37acb" exitCode=60 Oct 29 23:45:07.998356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.998267 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" event={"ID":"80e36a42-a6d4-449b-8923-f3a061fd58ec","Type":"ContainerDied","Data":"ce8d06d7ba0380cd16eff3b47ddb5060671b5ded3aa18d4f96c64a6c1de37acb"} Oct 29 23:45:07.998547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:07.998526 5736 scope.go:122] "RemoveContainer" containerID="ce8d06d7ba0380cd16eff3b47ddb5060671b5ded3aa18d4f96c64a6c1de37acb" Oct 29 23:45:07.998851 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:07.998828 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5_odh-ai-gateway-infra(80e36a42-a6d4-449b-8923-f3a061fd58ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" Oct 29 23:45:08.000333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.000311 5736 generic.go:356] "Generic (PLEG): container finished" podID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerID="e9e2dcc7d16d32e53218391658927434472eaf418e60e702b4d08d3dc2cd16d2" exitCode=60 Oct 29 23:45:08.000410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.000390 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" event={"ID":"1e5b2e09-ffc2-408c-a7a7-97b078c2984e","Type":"ContainerDied","Data":"e9e2dcc7d16d32e53218391658927434472eaf418e60e702b4d08d3dc2cd16d2"} Oct 29 23:45:08.000633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.000616 5736 scope.go:122] "RemoveContainer" containerID="e9e2dcc7d16d32e53218391658927434472eaf418e60e702b4d08d3dc2cd16d2" Oct 29 23:45:08.000847 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:08.000829 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x_odh-ai-gateway-infra(1e5b2e09-ffc2-408c-a7a7-97b078c2984e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" Oct 29 23:45:08.002294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.002276 5736 scope.go:122] "RemoveContainer" containerID="3f2bf5fb96debce9e2b80f27232812051f16a422ffd2b10df1a18478f0770df5" Oct 29 23:45:08.002537 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:08.002512 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth_odh-ai-gateway-infra(db376677-f21f-4807-ab75-b82ce5e8f51b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" Oct 29 23:45:08.004309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.004290 5736 generic.go:356] "Generic (PLEG): container finished" podID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerID="51c0e40b09398c2c99e4b1f1b64e2dd1abad50b5dcf6e53c5a9f4637ea735b59" exitCode=60 Oct 29 23:45:08.004404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.004373 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" event={"ID":"74c99f0d-50d0-4226-b5b7-fd1175bedb56","Type":"ContainerDied","Data":"51c0e40b09398c2c99e4b1f1b64e2dd1abad50b5dcf6e53c5a9f4637ea735b59"} Oct 29 23:45:08.004650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.004631 5736 scope.go:122] "RemoveContainer" containerID="51c0e40b09398c2c99e4b1f1b64e2dd1abad50b5dcf6e53c5a9f4637ea735b59" Oct 29 23:45:08.004917 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:08.004882 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz_odh-ai-gateway-infra(74c99f0d-50d0-4226-b5b7-fd1175bedb56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" Oct 29 23:45:08.006663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.006641 5736 scope.go:122] "RemoveContainer" containerID="dfc93df4c6cb6dd2c7d0594f4e431604fc88c7477bc661d82d01da8ed6886f52" Oct 29 23:45:08.006923 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:08.006904 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr_odh-ai-gateway-infra(5a37864a-0480-4fe9-b5c4-c946c1f049a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" Oct 29 23:45:08.008203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.008185 5736 scope.go:122] "RemoveContainer" containerID="1e8260565a096ef3eb110e7ac2efaaea977716d63373ade0b896fcbf613040dd" Oct 29 23:45:08.008474 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:08.008448 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8_odh-ai-gateway-infra(4cb9077b-6b07-4a72-907e-b3e9b0e2534a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" Oct 29 23:45:08.009931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.009911 5736 generic.go:356] "Generic (PLEG): container finished" podID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerID="79b67cf8f1b06e0375da141a7d4d84cba9648bb3e8a4de633404fa82a58bf587" exitCode=60 Oct 29 23:45:08.010019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.009994 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" event={"ID":"8c9e79f9-8c22-4a9f-87db-bd6413409296","Type":"ContainerDied","Data":"79b67cf8f1b06e0375da141a7d4d84cba9648bb3e8a4de633404fa82a58bf587"} Oct 29 23:45:08.010257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.010239 5736 scope.go:122] "RemoveContainer" containerID="79b67cf8f1b06e0375da141a7d4d84cba9648bb3e8a4de633404fa82a58bf587" Oct 29 23:45:08.010496 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:08.010470 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4_odh-ai-gateway-infra(8c9e79f9-8c22-4a9f-87db-bd6413409296)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" Oct 29 23:45:08.011884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.011864 5736 generic.go:356] "Generic (PLEG): container finished" podID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerID="124019b61304223e01bd7a0ac6f8824ea3911111cb02f99dc37a5bf6285e0f4e" exitCode=60 Oct 29 23:45:08.011971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.011913 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" event={"ID":"bd3eb218-be48-4f91-b4ca-94675bca35f5","Type":"ContainerDied","Data":"124019b61304223e01bd7a0ac6f8824ea3911111cb02f99dc37a5bf6285e0f4e"} Oct 29 23:45:08.012134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.012118 5736 scope.go:122] "RemoveContainer" containerID="124019b61304223e01bd7a0ac6f8824ea3911111cb02f99dc37a5bf6285e0f4e" Oct 29 23:45:08.012330 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:08.012312 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4_odh-ai-gateway-infra(bd3eb218-be48-4f91-b4ca-94675bca35f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" Oct 29 23:45:08.015617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.015598 5736 scope.go:122] "RemoveContainer" containerID="809982cd45b1ce87d6453a0c8bb6e71a6d1fd3043c6100a29dc7b4fde731ea7e" Oct 29 23:45:08.032170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.032141 5736 scope.go:122] "RemoveContainer" containerID="1e66057564c295bd2ddbffda34a669e15886e35e9490557f667e32ebe4f00a1a" Oct 29 23:45:08.047938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.047909 5736 scope.go:122] "RemoveContainer" containerID="92058f51033fe60c4799935e8db6780100b848b936e822d5b2800827013e617d" Oct 29 23:45:08.069279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.069250 5736 scope.go:122] "RemoveContainer" containerID="70b9b912f22f4749fd23cdfcaa72c2c3334c33030795b99f41654392d3fdadef" Oct 29 23:45:08.086762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.086740 5736 scope.go:122] "RemoveContainer" containerID="d46e389f6df5308d42f3cf6726088ef6956602ad115fc83416089eaaed4331a4" Oct 29 23:45:08.102760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.102733 5736 scope.go:122] "RemoveContainer" containerID="f91aee62e6fdff4aa920ff02221113ade0e313c2c4aead42909ffd695d729481" Oct 29 23:45:08.118671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.118645 5736 scope.go:122] "RemoveContainer" containerID="e005be50b08f3b6e5176cb09eb616ef5e83a6f6686818a041f4a30f322cd1982" Oct 29 23:45:08.136810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.136780 5736 scope.go:122] "RemoveContainer" containerID="d79c01d3619d1a0025e8d0528d198d8247768a2e02b41fabe9e8b75c38ab6db8" Oct 29 23:45:08.155917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.155887 5736 scope.go:122] "RemoveContainer" containerID="506fd7c827b205357e6e1b912158f7750b4f17830cceacf509f23eccd81ace6b" Oct 29 23:45:08.187462 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:08.187426 5736 scope.go:122] "RemoveContainer" containerID="7ab1fda15532c6b98ec4c4a66a5457b51658ad198b3e5309d6ea054841592cb6" Oct 29 23:45:09.023517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.023467 5736 scope.go:122] "RemoveContainer" containerID="ce8d06d7ba0380cd16eff3b47ddb5060671b5ded3aa18d4f96c64a6c1de37acb" Oct 29 23:45:09.023969 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.023915 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5_odh-ai-gateway-infra(80e36a42-a6d4-449b-8923-f3a061fd58ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" Oct 29 23:45:09.025479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.025452 5736 scope.go:122] "RemoveContainer" containerID="e9e2dcc7d16d32e53218391658927434472eaf418e60e702b4d08d3dc2cd16d2" Oct 29 23:45:09.025909 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.025881 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x_odh-ai-gateway-infra(1e5b2e09-ffc2-408c-a7a7-97b078c2984e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" Oct 29 23:45:09.027182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.027159 5736 scope.go:122] "RemoveContainer" containerID="51c0e40b09398c2c99e4b1f1b64e2dd1abad50b5dcf6e53c5a9f4637ea735b59" Oct 29 23:45:09.027520 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.027493 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz_odh-ai-gateway-infra(74c99f0d-50d0-4226-b5b7-fd1175bedb56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" Oct 29 23:45:09.028886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.028864 5736 scope.go:122] "RemoveContainer" containerID="79b67cf8f1b06e0375da141a7d4d84cba9648bb3e8a4de633404fa82a58bf587" Oct 29 23:45:09.029158 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.029134 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4_odh-ai-gateway-infra(8c9e79f9-8c22-4a9f-87db-bd6413409296)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" Oct 29 23:45:09.030544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.030525 5736 scope.go:122] "RemoveContainer" containerID="124019b61304223e01bd7a0ac6f8824ea3911111cb02f99dc37a5bf6285e0f4e" Oct 29 23:45:09.030850 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.030821 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4_odh-ai-gateway-infra(bd3eb218-be48-4f91-b4ca-94675bca35f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" Oct 29 23:45:09.032506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.032487 5736 scope.go:122] "RemoveContainer" containerID="7ebc6519b6da517c5cad02e2aa463672195583bbc742d1e233685d390240411e" Oct 29 23:45:09.032823 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.032796 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj_odh-ai-gateway-infra(aacb0496-adc4-4859-afe2-88a8c07d27b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" Oct 29 23:45:09.034463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.034442 5736 scope.go:122] "RemoveContainer" containerID="de776c564ffa0701f5cfbecd2e3cfefa24d39c8125cd83bc0952e5746a1ab4b5" Oct 29 23:45:09.034820 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.034795 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr_odh-ai-gateway-infra(137de700-2980-41fc-b778-24aa711d2ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" podUID="137de700-2980-41fc-b778-24aa711d2ac2" Oct 29 23:45:09.036496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.036337 5736 scope.go:122] "RemoveContainer" containerID="03d5d714d8fab3c0c20a154384fcb006372bbedf7fc8fd50c252cc1585ddae0b" Oct 29 23:45:09.036650 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.036628 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg_odh-ai-gateway-infra(7055786b-a7d7-4894-81c3-65ac4348fea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" Oct 29 23:45:09.038051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.038031 5736 scope.go:122] "RemoveContainer" containerID="991309b8643dd62e1ac9ec462059c6b50731b0a9142c1feede87d9b53abd47fe" Oct 29 23:45:09.038316 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.038280 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27_odh-ai-gateway-infra(fefefc63-6416-4fec-b277-6a8b9e9de6fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" Oct 29 23:45:09.039816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.039787 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" event={"ID":"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338","Type":"ContainerStarted","Data":"dd5adbcb5fe5786c8b14553cde8202328abe98734556b51c0ff94f9477f02e37"} Oct 29 23:45:09.041967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.041948 5736 scope.go:122] "RemoveContainer" containerID="dc82987dea9e983ac77edc31f38398e51b5a4a17fa607451b9544de9de0479f8" Oct 29 23:45:09.042274 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.042208 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk_odh-ai-gateway-infra(b61d2aa1-55ea-46a6-adbd-f7a03cf883fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" Oct 29 23:45:09.045809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.045789 5736 scope.go:122] "RemoveContainer" containerID="babedd511fe15a356eedd3768cbc91f5b8c607b0b05dc13be91909205395985b" Oct 29 23:45:09.046076 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:09.046055 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858_odh-ai-gateway-infra(bde02fb5-67b1-4da5-9dc3-63bc2b930f08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" Oct 29 23:45:09.111357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:09.111297 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" podStartSLOduration=3.111284314 podStartE2EDuration="3.111284314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:09.110254663 +0000 UTC m=+3706.226174268" watchObservedRunningTime="2026-10-29 23:45:09.111284314 +0000 UTC m=+3706.227203921" Oct 29 23:45:10.059201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:10.059164 5736 generic.go:356] "Generic (PLEG): container finished" podID="9b94d6d5-7d12-4c1e-9a2d-060b12d6e338" containerID="dd5adbcb5fe5786c8b14553cde8202328abe98734556b51c0ff94f9477f02e37" exitCode=0 Oct 29 23:45:10.059727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:10.059228 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" event={"ID":"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338","Type":"ContainerDied","Data":"dd5adbcb5fe5786c8b14553cde8202328abe98734556b51c0ff94f9477f02e37"} Oct 29 23:45:11.269069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.269042 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:11.405661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.394402 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-openshift-ca\") pod \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " Oct 29 23:45:11.405661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.394560 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-kube-api-access-kvwcb\" (UniqueName: \"kubernetes.io/projected/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-kube-api-access-kvwcb\") pod \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " Oct 29 23:45:11.405661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.394791 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-xks-ca\") pod \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\" (UID: \"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338\") " Oct 29 23:45:11.418611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.413853 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-openshift-ca" pod "9b94d6d5-7d12-4c1e-9a2d-060b12d6e338" (UID: "9b94d6d5-7d12-4c1e-9a2d-060b12d6e338"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:11.498042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.497991 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-xks-ca" pod "9b94d6d5-7d12-4c1e-9a2d-060b12d6e338" (UID: "9b94d6d5-7d12-4c1e-9a2d-060b12d6e338"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:11.594747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.588528 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-kube-api-access-kvwcb" pod "9b94d6d5-7d12-4c1e-9a2d-060b12d6e338" (UID: "9b94d6d5-7d12-4c1e-9a2d-060b12d6e338"). InnerVolumeSpecName "kube-api-access-kvwcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:11.656188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.656080 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:11.656188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.656125 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvwcb\" (UniqueName: \"kubernetes.io/projected/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-kube-api-access-kvwcb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:11.656188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:11.656141 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:12.079045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:12.079015 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" Oct 29 23:45:12.079230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:12.079011 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h" event={"ID":"9b94d6d5-7d12-4c1e-9a2d-060b12d6e338","Type":"ContainerDied","Data":"ee867950028eee512f867f4bddea2eedb6c2b12491d9a033e1f26150b6712b4b"} Oct 29 23:45:12.079230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:12.079122 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee867950028eee512f867f4bddea2eedb6c2b12491d9a033e1f26150b6712b4b" Oct 29 23:45:15.996742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:15.996709 5736 scope.go:122] "RemoveContainer" containerID="3f2bf5fb96debce9e2b80f27232812051f16a422ffd2b10df1a18478f0770df5" Oct 29 23:45:17.003948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.003905 5736 scope.go:122] "RemoveContainer" containerID="dc82987dea9e983ac77edc31f38398e51b5a4a17fa607451b9544de9de0479f8" Oct 29 23:45:17.004476 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.004076 5736 scope.go:122] "RemoveContainer" containerID="7623efecc41e6bc1b586cdfe6496550c6d0e64a6dd5f747e6bb2760c7cadcc4f" Oct 29 23:45:17.007122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.006095 5736 scope.go:122] "RemoveContainer" containerID="b394b888f76457d24a4ee6c9b8255291a2cc15c8a01335e33bd25412116e4d4f" Oct 29 23:45:17.030164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.029613 5736 scope.go:122] "RemoveContainer" containerID="991309b8643dd62e1ac9ec462059c6b50731b0a9142c1feede87d9b53abd47fe" Oct 29 23:45:17.036435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.035820 5736 scope.go:122] "RemoveContainer" containerID="7ebc6519b6da517c5cad02e2aa463672195583bbc742d1e233685d390240411e" Oct 29 23:45:17.137158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.137127 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" event={"ID":"db376677-f21f-4807-ab75-b82ce5e8f51b","Type":"ContainerStarted","Data":"a1b6ff188cb3a9c2906465f5a012c491fe3eb553b4c49cc44055aa40543ca6d6"} Oct 29 23:45:17.322959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.322833 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq"] Oct 29 23:45:17.329382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.329342 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b73a75-vcstq"] Oct 29 23:45:17.996650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.996575 5736 scope.go:122] "RemoveContainer" containerID="babedd511fe15a356eedd3768cbc91f5b8c607b0b05dc13be91909205395985b" Oct 29 23:45:17.996873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.996753 5736 scope.go:122] "RemoveContainer" containerID="dfc93df4c6cb6dd2c7d0594f4e431604fc88c7477bc661d82d01da8ed6886f52" Oct 29 23:45:17.996964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.996895 5736 scope.go:122] "RemoveContainer" containerID="1e8260565a096ef3eb110e7ac2efaaea977716d63373ade0b896fcbf613040dd" Oct 29 23:45:17.997027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.997004 5736 scope.go:122] "RemoveContainer" containerID="5e24234f0dd68d39474472677ce16deb35e4d6e23b1fd9cf6855c429d4e67eed" Oct 29 23:45:17.997297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:17.997271 5736 scope.go:122] "RemoveContainer" containerID="de776c564ffa0701f5cfbecd2e3cfefa24d39c8125cd83bc0952e5746a1ab4b5" Oct 29 23:45:18.157549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.157504 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" event={"ID":"dd63cd40-3af1-49cb-b17b-5576a1f434cd","Type":"ContainerStarted","Data":"3ace84283b60d0afb3430e32a2b4c94d11055d4d45754e9d0a6588c80a15b06b"} Oct 29 23:45:18.159655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.159619 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" event={"ID":"aacb0496-adc4-4859-afe2-88a8c07d27b3","Type":"ContainerStarted","Data":"acbee514c9b70b969ac7346c5b2adb93b97c57fb210a3a17edb6c98168d58ad2"} Oct 29 23:45:18.161614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.161559 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" event={"ID":"fefefc63-6416-4fec-b277-6a8b9e9de6fc","Type":"ContainerStarted","Data":"6f227411b10ce9499a7ebef9c9f2350eb5cf5882ec2bed606c73aaabcd6de3b6"} Oct 29 23:45:18.163596 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.163561 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" event={"ID":"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa","Type":"ContainerStarted","Data":"e83aedbbe863a7f7738a71d96a8a6bb15396d4c52dc30e8e78a14046e6eb4f21"} Oct 29 23:45:18.165786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.165760 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" event={"ID":"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb","Type":"ContainerStarted","Data":"8fdfcedb63e92a1971495aeef578dacaba6a3dac16b0afc16501bbef41682f02"} Oct 29 23:45:18.167883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.167852 5736 generic.go:356] "Generic (PLEG): container finished" podID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerID="a1b6ff188cb3a9c2906465f5a012c491fe3eb553b4c49cc44055aa40543ca6d6" exitCode=60 Oct 29 23:45:18.167998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.167889 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" event={"ID":"db376677-f21f-4807-ab75-b82ce5e8f51b","Type":"ContainerDied","Data":"a1b6ff188cb3a9c2906465f5a012c491fe3eb553b4c49cc44055aa40543ca6d6"} Oct 29 23:45:18.167998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.167938 5736 scope.go:122] "RemoveContainer" containerID="3f2bf5fb96debce9e2b80f27232812051f16a422ffd2b10df1a18478f0770df5" Oct 29 23:45:18.168322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.168301 5736 scope.go:122] "RemoveContainer" containerID="a1b6ff188cb3a9c2906465f5a012c491fe3eb553b4c49cc44055aa40543ca6d6" Oct 29 23:45:18.168649 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:18.168622 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth_odh-ai-gateway-infra(db376677-f21f-4807-ab75-b82ce5e8f51b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" Oct 29 23:45:18.194709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.194670 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth"] Oct 29 23:45:18.214520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.207700 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" podStartSLOduration=18.207683288 podStartE2EDuration="18.207683288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:18.199820238 +0000 UTC m=+3715.315739838" watchObservedRunningTime="2026-10-29 23:45:18.207683288 +0000 UTC m=+3715.323602894" Oct 29 23:45:18.617117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.617082 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n"] Oct 29 23:45:18.655793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.655762 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw"] Oct 29 23:45:18.996401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.996370 5736 scope.go:122] "RemoveContainer" containerID="79b67cf8f1b06e0375da141a7d4d84cba9648bb3e8a4de633404fa82a58bf587" Oct 29 23:45:18.998100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:18.998051 5736 scope.go:122] "RemoveContainer" containerID="ce8d06d7ba0380cd16eff3b47ddb5060671b5ded3aa18d4f96c64a6c1de37acb" Oct 29 23:45:19.012822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.012792 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c71943-9377-4161-bb98-8850f7f14f97" path="/var/lib/kubelet/pods/d4c71943-9377-4161-bb98-8850f7f14f97/volumes" Oct 29 23:45:19.180032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.180001 5736 generic.go:356] "Generic (PLEG): container finished" podID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerID="8fdfcedb63e92a1971495aeef578dacaba6a3dac16b0afc16501bbef41682f02" exitCode=60 Oct 29 23:45:19.180401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.180069 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" event={"ID":"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb","Type":"ContainerDied","Data":"8fdfcedb63e92a1971495aeef578dacaba6a3dac16b0afc16501bbef41682f02"} Oct 29 23:45:19.180401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.180125 5736 scope.go:122] "RemoveContainer" containerID="dc82987dea9e983ac77edc31f38398e51b5a4a17fa607451b9544de9de0479f8" Oct 29 23:45:19.180644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.180620 5736 scope.go:122] "RemoveContainer" containerID="8fdfcedb63e92a1971495aeef578dacaba6a3dac16b0afc16501bbef41682f02" Oct 29 23:45:19.181452 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:19.181078 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk_odh-ai-gateway-infra(b61d2aa1-55ea-46a6-adbd-f7a03cf883fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" Oct 29 23:45:19.182366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.182342 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" event={"ID":"52ff5a60-2de6-41b0-b7c9-0864c9086318","Type":"ContainerStarted","Data":"3970be43072cf62fbfe60e5ab75d4b63cbfb699b769e7912ef873f6435dffb42"} Oct 29 23:45:19.184398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.184374 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" event={"ID":"bde02fb5-67b1-4da5-9dc3-63bc2b930f08","Type":"ContainerStarted","Data":"a3ec2dbc07510300b5c81702e18fb29172c585932776a353ef98a27d99d0db3c"} Oct 29 23:45:19.188496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.188472 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" event={"ID":"5a37864a-0480-4fe9-b5c4-c946c1f049a9","Type":"ContainerStarted","Data":"939011f0ca24fa8d3885fa6679507176af9a62a5894e7576b2bb0affd694c010"} Oct 29 23:45:19.191119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.191093 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" event={"ID":"4cb9077b-6b07-4a72-907e-b3e9b0e2534a","Type":"ContainerStarted","Data":"20738a2be0d2525d945e67464f9e3d2a22a990ca194d039a18a73b1ff2d9b8a9"} Oct 29 23:45:19.193381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.193359 5736 generic.go:356] "Generic (PLEG): container finished" podID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerID="3ace84283b60d0afb3430e32a2b4c94d11055d4d45754e9d0a6588c80a15b06b" exitCode=60 Oct 29 23:45:19.193514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.193428 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" event={"ID":"dd63cd40-3af1-49cb-b17b-5576a1f434cd","Type":"ContainerDied","Data":"3ace84283b60d0afb3430e32a2b4c94d11055d4d45754e9d0a6588c80a15b06b"} Oct 29 23:45:19.193806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.193784 5736 scope.go:122] "RemoveContainer" containerID="3ace84283b60d0afb3430e32a2b4c94d11055d4d45754e9d0a6588c80a15b06b" Oct 29 23:45:19.194107 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:19.194082 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4_odh-ai-gateway-infra(dd63cd40-3af1-49cb-b17b-5576a1f434cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" Oct 29 23:45:19.195507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.195483 5736 generic.go:356] "Generic (PLEG): container finished" podID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerID="acbee514c9b70b969ac7346c5b2adb93b97c57fb210a3a17edb6c98168d58ad2" exitCode=60 Oct 29 23:45:19.195756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.195550 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" event={"ID":"aacb0496-adc4-4859-afe2-88a8c07d27b3","Type":"ContainerDied","Data":"acbee514c9b70b969ac7346c5b2adb93b97c57fb210a3a17edb6c98168d58ad2"} Oct 29 23:45:19.196009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.195964 5736 scope.go:122] "RemoveContainer" containerID="acbee514c9b70b969ac7346c5b2adb93b97c57fb210a3a17edb6c98168d58ad2" Oct 29 23:45:19.196309 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:19.196280 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj_odh-ai-gateway-infra(aacb0496-adc4-4859-afe2-88a8c07d27b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" Oct 29 23:45:19.198635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.198607 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" event={"ID":"137de700-2980-41fc-b778-24aa711d2ac2","Type":"ContainerStarted","Data":"334283a1c879efc18413746784447ba84e18f7c9b8d10de6bf684af7be34d620"} Oct 29 23:45:19.200746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.200722 5736 scope.go:122] "RemoveContainer" containerID="7623efecc41e6bc1b586cdfe6496550c6d0e64a6dd5f747e6bb2760c7cadcc4f" Oct 29 23:45:19.201186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.201153 5736 generic.go:356] "Generic (PLEG): container finished" podID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerID="6f227411b10ce9499a7ebef9c9f2350eb5cf5882ec2bed606c73aaabcd6de3b6" exitCode=60 Oct 29 23:45:19.201349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.201203 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" event={"ID":"fefefc63-6416-4fec-b277-6a8b9e9de6fc","Type":"ContainerDied","Data":"6f227411b10ce9499a7ebef9c9f2350eb5cf5882ec2bed606c73aaabcd6de3b6"} Oct 29 23:45:19.201621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.201557 5736 scope.go:122] "RemoveContainer" containerID="6f227411b10ce9499a7ebef9c9f2350eb5cf5882ec2bed606c73aaabcd6de3b6" Oct 29 23:45:19.201941 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:19.201914 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27_odh-ai-gateway-infra(fefefc63-6416-4fec-b277-6a8b9e9de6fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" Oct 29 23:45:19.208312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.204369 5736 generic.go:356] "Generic (PLEG): container finished" podID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerID="e83aedbbe863a7f7738a71d96a8a6bb15396d4c52dc30e8e78a14046e6eb4f21" exitCode=60 Oct 29 23:45:19.208312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.204467 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" event={"ID":"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa","Type":"ContainerDied","Data":"e83aedbbe863a7f7738a71d96a8a6bb15396d4c52dc30e8e78a14046e6eb4f21"} Oct 29 23:45:19.208312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.204943 5736 scope.go:122] "RemoveContainer" containerID="e83aedbbe863a7f7738a71d96a8a6bb15396d4c52dc30e8e78a14046e6eb4f21" Oct 29 23:45:19.208312 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:19.205186 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct_odh-ai-gateway-infra(bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" Oct 29 23:45:19.218812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.218777 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4"] Oct 29 23:45:19.271775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.260306 5736 scope.go:122] "RemoveContainer" containerID="7ebc6519b6da517c5cad02e2aa463672195583bbc742d1e233685d390240411e" Oct 29 23:45:19.728513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.728276 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj"] Oct 29 23:45:19.745976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.745951 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct"] Oct 29 23:45:19.760008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.759972 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27"] Oct 29 23:45:19.772541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.772503 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk"] Oct 29 23:45:19.818015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.817986 5736 scope.go:122] "RemoveContainer" containerID="991309b8643dd62e1ac9ec462059c6b50731b0a9142c1feede87d9b53abd47fe" Oct 29 23:45:19.825791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.825761 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:19.835525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.835497 5736 scope.go:122] "RemoveContainer" containerID="b394b888f76457d24a4ee6c9b8255291a2cc15c8a01335e33bd25412116e4d4f" Oct 29 23:45:19.991444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.991392 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db376677-f21f-4807-ab75-b82ce5e8f51b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db376677-f21f-4807-ab75-b82ce5e8f51b-maas-api-service-ca\") pod \"db376677-f21f-4807-ab75-b82ce5e8f51b\" (UID: \"db376677-f21f-4807-ab75-b82ce5e8f51b\") " Oct 29 23:45:19.991726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.991539 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db376677-f21f-4807-ab75-b82ce5e8f51b-kube-api-access-tq9fr\" (UniqueName: \"kubernetes.io/projected/db376677-f21f-4807-ab75-b82ce5e8f51b-kube-api-access-tq9fr\") pod \"db376677-f21f-4807-ab75-b82ce5e8f51b\" (UID: \"db376677-f21f-4807-ab75-b82ce5e8f51b\") " Oct 29 23:45:19.991836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:19.991808 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db376677-f21f-4807-ab75-b82ce5e8f51b-maas-api-service-ca" pod "db376677-f21f-4807-ab75-b82ce5e8f51b" (UID: "db376677-f21f-4807-ab75-b82ce5e8f51b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.001282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.001215 5736 scope.go:122] "RemoveContainer" containerID="03d5d714d8fab3c0c20a154384fcb006372bbedf7fc8fd50c252cc1585ddae0b" Oct 29 23:45:20.005049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.002210 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db376677-f21f-4807-ab75-b82ce5e8f51b-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.005049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.003059 5736 scope.go:122] "RemoveContainer" containerID="124019b61304223e01bd7a0ac6f8824ea3911111cb02f99dc37a5bf6285e0f4e" Oct 29 23:45:20.005049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.003748 5736 scope.go:122] "RemoveContainer" containerID="51c0e40b09398c2c99e4b1f1b64e2dd1abad50b5dcf6e53c5a9f4637ea735b59" Oct 29 23:45:20.005049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.004551 5736 scope.go:122] "RemoveContainer" containerID="f92f51093d4bc19624ae5a30f27814b2a4a0ae64f2d32890c50baa8b942525e9" Oct 29 23:45:20.005549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.005259 5736 scope.go:122] "RemoveContainer" containerID="e9e2dcc7d16d32e53218391658927434472eaf418e60e702b4d08d3dc2cd16d2" Oct 29 23:45:20.095678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.095418 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db376677-f21f-4807-ab75-b82ce5e8f51b-kube-api-access-tq9fr" pod "db376677-f21f-4807-ab75-b82ce5e8f51b" (UID: "db376677-f21f-4807-ab75-b82ce5e8f51b"). InnerVolumeSpecName "kube-api-access-tq9fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.109144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.109094 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9fr\" (UniqueName: \"kubernetes.io/projected/db376677-f21f-4807-ab75-b82ce5e8f51b-kube-api-access-tq9fr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.248837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.248799 5736 generic.go:356] "Generic (PLEG): container finished" podID="137de700-2980-41fc-b778-24aa711d2ac2" containerID="334283a1c879efc18413746784447ba84e18f7c9b8d10de6bf684af7be34d620" exitCode=60 Oct 29 23:45:20.249146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.248921 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" event={"ID":"137de700-2980-41fc-b778-24aa711d2ac2","Type":"ContainerDied","Data":"334283a1c879efc18413746784447ba84e18f7c9b8d10de6bf684af7be34d620"} Oct 29 23:45:20.249146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.248986 5736 scope.go:122] "RemoveContainer" containerID="de776c564ffa0701f5cfbecd2e3cfefa24d39c8125cd83bc0952e5746a1ab4b5" Oct 29 23:45:20.249566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.249542 5736 scope.go:122] "RemoveContainer" containerID="334283a1c879efc18413746784447ba84e18f7c9b8d10de6bf684af7be34d620" Oct 29 23:45:20.250229 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:20.250089 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr_odh-ai-gateway-infra(137de700-2980-41fc-b778-24aa711d2ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" podUID="137de700-2980-41fc-b778-24aa711d2ac2" Oct 29 23:45:20.279673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.278863 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" event={"ID":"52ff5a60-2de6-41b0-b7c9-0864c9086318","Type":"ContainerDied","Data":"3970be43072cf62fbfe60e5ab75d4b63cbfb699b769e7912ef873f6435dffb42"} Oct 29 23:45:20.279673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.279456 5736 scope.go:122] "RemoveContainer" containerID="3970be43072cf62fbfe60e5ab75d4b63cbfb699b769e7912ef873f6435dffb42" Oct 29 23:45:20.280728 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:20.280335 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6_odh-ai-gateway-infra(52ff5a60-2de6-41b0-b7c9-0864c9086318)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" Oct 29 23:45:20.280901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.280877 5736 generic.go:356] "Generic (PLEG): container finished" podID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerID="3970be43072cf62fbfe60e5ab75d4b63cbfb699b769e7912ef873f6435dffb42" exitCode=60 Oct 29 23:45:20.284883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.283550 5736 scope.go:122] "RemoveContainer" containerID="5e24234f0dd68d39474472677ce16deb35e4d6e23b1fd9cf6855c429d4e67eed" Oct 29 23:45:20.292800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.292026 5736 generic.go:356] "Generic (PLEG): container finished" podID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerID="a3ec2dbc07510300b5c81702e18fb29172c585932776a353ef98a27d99d0db3c" exitCode=60 Oct 29 23:45:20.292800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.292178 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" event={"ID":"bde02fb5-67b1-4da5-9dc3-63bc2b930f08","Type":"ContainerDied","Data":"a3ec2dbc07510300b5c81702e18fb29172c585932776a353ef98a27d99d0db3c"} Oct 29 23:45:20.294182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.293533 5736 scope.go:122] "RemoveContainer" containerID="a3ec2dbc07510300b5c81702e18fb29172c585932776a353ef98a27d99d0db3c" Oct 29 23:45:20.299609 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:20.294731 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858_odh-ai-gateway-infra(bde02fb5-67b1-4da5-9dc3-63bc2b930f08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" Oct 29 23:45:20.372114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.366366 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" event={"ID":"80e36a42-a6d4-449b-8923-f3a061fd58ec","Type":"ContainerStarted","Data":"7c4217ebf5501ae3d45daef17c55ca479439654298025a4efd16f82b23dbab77"} Oct 29 23:45:20.372114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.368854 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" Oct 29 23:45:20.378600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.372655 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth" event={"ID":"db376677-f21f-4807-ab75-b82ce5e8f51b","Type":"ContainerDied","Data":"c8f3237a4c5bf2350165fa5aa1c74a03c5a92fafea366324bd19720fb422087d"} Oct 29 23:45:20.378600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.376259 5736 generic.go:356] "Generic (PLEG): container finished" podID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerID="939011f0ca24fa8d3885fa6679507176af9a62a5894e7576b2bb0affd694c010" exitCode=60 Oct 29 23:45:20.378600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.376364 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" event={"ID":"5a37864a-0480-4fe9-b5c4-c946c1f049a9","Type":"ContainerDied","Data":"939011f0ca24fa8d3885fa6679507176af9a62a5894e7576b2bb0affd694c010"} Oct 29 23:45:20.378600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.376934 5736 scope.go:122] "RemoveContainer" containerID="939011f0ca24fa8d3885fa6679507176af9a62a5894e7576b2bb0affd694c010" Oct 29 23:45:20.378600 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:20.377399 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr_odh-ai-gateway-infra(5a37864a-0480-4fe9-b5c4-c946c1f049a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" Oct 29 23:45:20.380642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.380073 5736 scope.go:122] "RemoveContainer" containerID="babedd511fe15a356eedd3768cbc91f5b8c607b0b05dc13be91909205395985b" Oct 29 23:45:20.382856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.382829 5736 generic.go:356] "Generic (PLEG): container finished" podID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerID="20738a2be0d2525d945e67464f9e3d2a22a990ca194d039a18a73b1ff2d9b8a9" exitCode=60 Oct 29 23:45:20.382978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.382884 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" event={"ID":"4cb9077b-6b07-4a72-907e-b3e9b0e2534a","Type":"ContainerDied","Data":"20738a2be0d2525d945e67464f9e3d2a22a990ca194d039a18a73b1ff2d9b8a9"} Oct 29 23:45:20.383873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.383301 5736 scope.go:122] "RemoveContainer" containerID="20738a2be0d2525d945e67464f9e3d2a22a990ca194d039a18a73b1ff2d9b8a9" Oct 29 23:45:20.383873 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:20.383818 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8_odh-ai-gateway-infra(4cb9077b-6b07-4a72-907e-b3e9b0e2534a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" Oct 29 23:45:20.411770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.406773 5736 scope.go:122] "RemoveContainer" containerID="a1b6ff188cb3a9c2906465f5a012c491fe3eb553b4c49cc44055aa40543ca6d6" Oct 29 23:45:20.427423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.427391 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" event={"ID":"8c9e79f9-8c22-4a9f-87db-bd6413409296","Type":"ContainerStarted","Data":"b3641258aac00f10b965560bf59a74bfdb9242972a6ec0c8393b04025e0d4436"} Oct 29 23:45:20.454952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.454922 5736 scope.go:122] "RemoveContainer" containerID="dfc93df4c6cb6dd2c7d0594f4e431604fc88c7477bc661d82d01da8ed6886f52" Oct 29 23:45:20.478390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.477806 5736 scope.go:122] "RemoveContainer" containerID="1e8260565a096ef3eb110e7ac2efaaea977716d63373ade0b896fcbf613040dd" Oct 29 23:45:20.685805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.685664 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth"] Oct 29 23:45:20.708741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.694307 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f162a-29888625-4trth"] Oct 29 23:45:20.724682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.724574 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:20.731192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.731155 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:20.743137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.743110 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:20.746018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.745993 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:20.748854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.748794 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:20.776243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.776209 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr"] Oct 29 23:45:20.827658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.827624 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd63cd40-3af1-49cb-b17b-5576a1f434cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63cd40-3af1-49cb-b17b-5576a1f434cd-maas-api-service-ca\") pod \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\" (UID: \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\") " Oct 29 23:45:20.827856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.827699 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-kube-api-access-jp665\" (UniqueName: \"kubernetes.io/projected/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-kube-api-access-jp665\") pod \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\" (UID: \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\") " Oct 29 23:45:20.827856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.827717 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacb0496-adc4-4859-afe2-88a8c07d27b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacb0496-adc4-4859-afe2-88a8c07d27b3-maas-api-service-ca\") pod \"aacb0496-adc4-4859-afe2-88a8c07d27b3\" (UID: \"aacb0496-adc4-4859-afe2-88a8c07d27b3\") " Oct 29 23:45:20.827997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.827857 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefefc63-6416-4fec-b277-6a8b9e9de6fc-kube-api-access-59g85\" (UniqueName: \"kubernetes.io/projected/fefefc63-6416-4fec-b277-6a8b9e9de6fc-kube-api-access-59g85\") pod \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\" (UID: \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\") " Oct 29 23:45:20.827997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.827927 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-kube-api-access-sgmbr\" (UniqueName: \"kubernetes.io/projected/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-kube-api-access-sgmbr\") pod \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\" (UID: \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\") " Oct 29 23:45:20.827997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.827972 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefefc63-6416-4fec-b277-6a8b9e9de6fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefefc63-6416-4fec-b277-6a8b9e9de6fc-maas-api-service-ca\") pod \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\" (UID: \"fefefc63-6416-4fec-b277-6a8b9e9de6fc\") " Oct 29 23:45:20.828183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.828022 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacb0496-adc4-4859-afe2-88a8c07d27b3-kube-api-access-rsftg\" (UniqueName: \"kubernetes.io/projected/aacb0496-adc4-4859-afe2-88a8c07d27b3-kube-api-access-rsftg\") pod \"aacb0496-adc4-4859-afe2-88a8c07d27b3\" (UID: \"aacb0496-adc4-4859-afe2-88a8c07d27b3\") " Oct 29 23:45:20.828183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.828092 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-maas-api-service-ca\") pod \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\" (UID: \"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb\") " Oct 29 23:45:20.828183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.828122 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd63cd40-3af1-49cb-b17b-5576a1f434cd-kube-api-access-tfhmk\" (UniqueName: \"kubernetes.io/projected/dd63cd40-3af1-49cb-b17b-5576a1f434cd-kube-api-access-tfhmk\") pod \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\" (UID: \"dd63cd40-3af1-49cb-b17b-5576a1f434cd\") " Oct 29 23:45:20.828393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.828247 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd63cd40-3af1-49cb-b17b-5576a1f434cd-maas-api-service-ca" pod "dd63cd40-3af1-49cb-b17b-5576a1f434cd" (UID: "dd63cd40-3af1-49cb-b17b-5576a1f434cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.829260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.828795 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-maas-api-service-ca\") pod \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\" (UID: \"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa\") " Oct 29 23:45:20.829260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.828888 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacb0496-adc4-4859-afe2-88a8c07d27b3-maas-api-service-ca" pod "aacb0496-adc4-4859-afe2-88a8c07d27b3" (UID: "aacb0496-adc4-4859-afe2-88a8c07d27b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.829260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.828976 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-maas-api-service-ca" pod "b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" (UID: "b61d2aa1-55ea-46a6-adbd-f7a03cf883fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.829260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.829215 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefefc63-6416-4fec-b277-6a8b9e9de6fc-maas-api-service-ca" pod "fefefc63-6416-4fec-b277-6a8b9e9de6fc" (UID: "fefefc63-6416-4fec-b277-6a8b9e9de6fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.830556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.829319 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-maas-api-service-ca" pod "bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" (UID: "bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:20.830556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.830309 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fefefc63-6416-4fec-b277-6a8b9e9de6fc-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.830556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.830332 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.830556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.830366 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.830556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.830381 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63cd40-3af1-49cb-b17b-5576a1f434cd-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.830556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.830395 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aacb0496-adc4-4859-afe2-88a8c07d27b3-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.834158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.834130 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-kube-api-access-jp665" pod "b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" (UID: "b61d2aa1-55ea-46a6-adbd-f7a03cf883fb"). InnerVolumeSpecName "kube-api-access-jp665". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.835270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.835232 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacb0496-adc4-4859-afe2-88a8c07d27b3-kube-api-access-rsftg" pod "aacb0496-adc4-4859-afe2-88a8c07d27b3" (UID: "aacb0496-adc4-4859-afe2-88a8c07d27b3"). InnerVolumeSpecName "kube-api-access-rsftg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.836690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.836569 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefefc63-6416-4fec-b277-6a8b9e9de6fc-kube-api-access-59g85" pod "fefefc63-6416-4fec-b277-6a8b9e9de6fc" (UID: "fefefc63-6416-4fec-b277-6a8b9e9de6fc"). InnerVolumeSpecName "kube-api-access-59g85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.837046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.837011 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd63cd40-3af1-49cb-b17b-5576a1f434cd-kube-api-access-tfhmk" pod "dd63cd40-3af1-49cb-b17b-5576a1f434cd" (UID: "dd63cd40-3af1-49cb-b17b-5576a1f434cd"). InnerVolumeSpecName "kube-api-access-tfhmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.837662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.837622 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-kube-api-access-sgmbr" pod "bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" (UID: "bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa"). InnerVolumeSpecName "kube-api-access-sgmbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:20.932343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.932200 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp665\" (UniqueName: \"kubernetes.io/projected/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb-kube-api-access-jp665\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.932343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.932235 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59g85\" (UniqueName: \"kubernetes.io/projected/fefefc63-6416-4fec-b277-6a8b9e9de6fc-kube-api-access-59g85\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.932343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.932249 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgmbr\" (UniqueName: \"kubernetes.io/projected/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa-kube-api-access-sgmbr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.932343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.932264 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsftg\" (UniqueName: \"kubernetes.io/projected/aacb0496-adc4-4859-afe2-88a8c07d27b3-kube-api-access-rsftg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.932343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.932278 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhmk\" (UniqueName: \"kubernetes.io/projected/dd63cd40-3af1-49cb-b17b-5576a1f434cd-kube-api-access-tfhmk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:20.984488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:20.984448 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr"] Oct 29 23:45:21.013595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.013011 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" path="/var/lib/kubelet/pods/db376677-f21f-4807-ab75-b82ce5e8f51b/volumes" Oct 29 23:45:21.034178 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.034149 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8"] Oct 29 23:45:21.072423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.072383 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858"] Oct 29 23:45:21.163095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.163035 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6"] Oct 29 23:45:21.441704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.441613 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" event={"ID":"dd63cd40-3af1-49cb-b17b-5576a1f434cd","Type":"ContainerDied","Data":"656e37fd37d036fb9a343c58869926c47396b553d2d7dd43a50239ef11da9143"} Oct 29 23:45:21.441704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.441649 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4" Oct 29 23:45:21.441704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.441698 5736 scope.go:122] "RemoveContainer" containerID="3ace84283b60d0afb3430e32a2b4c94d11055d4d45754e9d0a6588c80a15b06b" Oct 29 23:45:21.443361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.443335 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" event={"ID":"aacb0496-adc4-4859-afe2-88a8c07d27b3","Type":"ContainerDied","Data":"af4a4961da538bd7b1a2e926674d6a3ab4dc6c116c71bf5193cd2628ec0a9d35"} Oct 29 23:45:21.443800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.443353 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj" Oct 29 23:45:21.446207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.446178 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" event={"ID":"66794c35-2684-4c32-a7c4-7df95beb1b5c","Type":"ContainerStarted","Data":"595f50f203140d89c558326180e6b922b9e880991e601cf8e3fc22911d5bda9d"} Oct 29 23:45:21.451679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.451649 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" event={"ID":"7055786b-a7d7-4894-81c3-65ac4348fea9","Type":"ContainerStarted","Data":"da39cbfc29bd1ba8121d97e1fe69b35700cfe3aed17f9e5709ef18b0621ac040"} Oct 29 23:45:21.453965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.453937 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" event={"ID":"fefefc63-6416-4fec-b277-6a8b9e9de6fc","Type":"ContainerDied","Data":"7abcbd61b058ddb5b5cb80a66d7b13971cd01841dfd982d62dfb8494a75e2e52"} Oct 29 23:45:21.454124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.454045 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27" Oct 29 23:45:21.457320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.457295 5736 scope.go:122] "RemoveContainer" containerID="acbee514c9b70b969ac7346c5b2adb93b97c57fb210a3a17edb6c98168d58ad2" Oct 29 23:45:21.457689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.457663 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" event={"ID":"bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa","Type":"ContainerDied","Data":"ec3c62eae3465baa38bcd3f46bfaca5a8da585dd67910886cad942cea65514b1"} Oct 29 23:45:21.457798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.457762 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct" Oct 29 23:45:21.459948 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.459917 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" event={"ID":"b61d2aa1-55ea-46a6-adbd-f7a03cf883fb","Type":"ContainerDied","Data":"f64de5aafa0c561351c2391af5441b3d6e8872df4a85936a87fdf42ab7bb0b67"} Oct 29 23:45:21.460218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.460031 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk" Oct 29 23:45:21.469900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.469870 5736 generic.go:356] "Generic (PLEG): container finished" podID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerID="7c4217ebf5501ae3d45daef17c55ca479439654298025a4efd16f82b23dbab77" exitCode=60 Oct 29 23:45:21.470048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.469964 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" event={"ID":"80e36a42-a6d4-449b-8923-f3a061fd58ec","Type":"ContainerDied","Data":"7c4217ebf5501ae3d45daef17c55ca479439654298025a4efd16f82b23dbab77"} Oct 29 23:45:21.470563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.470537 5736 scope.go:122] "RemoveContainer" containerID="7c4217ebf5501ae3d45daef17c55ca479439654298025a4efd16f82b23dbab77" Oct 29 23:45:21.471064 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:21.471040 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5_odh-ai-gateway-infra(80e36a42-a6d4-449b-8923-f3a061fd58ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" Oct 29 23:45:21.472918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.472893 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" event={"ID":"1e5b2e09-ffc2-408c-a7a7-97b078c2984e","Type":"ContainerStarted","Data":"6bce52f5b27674cec7b70b9f0e22ef7af6a20ddc710a23235f672e55d98cc50e"} Oct 29 23:45:21.477433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.476669 5736 scope.go:122] "RemoveContainer" containerID="6f227411b10ce9499a7ebef9c9f2350eb5cf5882ec2bed606c73aaabcd6de3b6" Oct 29 23:45:21.483262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.483199 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" event={"ID":"74c99f0d-50d0-4226-b5b7-fd1175bedb56","Type":"ContainerStarted","Data":"a0a52544de6389f79aba33a5c71764e9ee85ff0fdfd3e91a12f88fddb592f2df"} Oct 29 23:45:21.556192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.550822 5736 scope.go:122] "RemoveContainer" containerID="e83aedbbe863a7f7738a71d96a8a6bb15396d4c52dc30e8e78a14046e6eb4f21" Oct 29 23:45:21.605230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.604946 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj"] Oct 29 23:45:21.614519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.614492 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-32c192-29888625-7dsgj"] Oct 29 23:45:21.627651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.626047 5736 generic.go:356] "Generic (PLEG): container finished" podID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerID="b3641258aac00f10b965560bf59a74bfdb9242972a6ec0c8393b04025e0d4436" exitCode=60 Oct 29 23:45:21.627651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.626201 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" event={"ID":"8c9e79f9-8c22-4a9f-87db-bd6413409296","Type":"ContainerDied","Data":"b3641258aac00f10b965560bf59a74bfdb9242972a6ec0c8393b04025e0d4436"} Oct 29 23:45:21.627651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.626753 5736 scope.go:122] "RemoveContainer" containerID="b3641258aac00f10b965560bf59a74bfdb9242972a6ec0c8393b04025e0d4436" Oct 29 23:45:21.627651 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:21.627231 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4_odh-ai-gateway-infra(8c9e79f9-8c22-4a9f-87db-bd6413409296)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" Oct 29 23:45:21.631048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.631020 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" event={"ID":"bd3eb218-be48-4f91-b4ca-94675bca35f5","Type":"ContainerStarted","Data":"8091bcf6056ee5a19af154cf5dfb34b960f947aa44ec5015d6b963b460d28927"} Oct 29 23:45:21.633272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.631600 5736 scope.go:122] "RemoveContainer" containerID="8091bcf6056ee5a19af154cf5dfb34b960f947aa44ec5015d6b963b460d28927" Oct 29 23:45:21.633272 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:21.632819 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4_odh-ai-gateway-infra(bd3eb218-be48-4f91-b4ca-94675bca35f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" Oct 29 23:45:21.737327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.737051 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4"] Oct 29 23:45:21.740411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.740379 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e59cd-29888625-r5lp4"] Oct 29 23:45:21.802044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.801997 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5"] Oct 29 23:45:21.886262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.886227 5736 scope.go:122] "RemoveContainer" containerID="8fdfcedb63e92a1971495aeef578dacaba6a3dac16b0afc16501bbef41682f02" Oct 29 23:45:21.897077 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.897048 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:21.902155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.902123 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4"] Oct 29 23:45:21.907127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.907101 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:21.911715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.911688 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:21.912533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.912513 5736 scope.go:122] "RemoveContainer" containerID="ce8d06d7ba0380cd16eff3b47ddb5060671b5ded3aa18d4f96c64a6c1de37acb" Oct 29 23:45:21.923509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.923485 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:21.927360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.927327 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" podUID="edc14cbd-79a0-449f-a896-eda15ab761c6" containerName="main" containerID="cri-o://ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171" gracePeriod=45 Oct 29 23:45:21.928074 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.928049 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:22.011335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.967341 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137de700-2980-41fc-b778-24aa711d2ac2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137de700-2980-41fc-b778-24aa711d2ac2-maas-api-service-ca\") pod \"137de700-2980-41fc-b778-24aa711d2ac2\" (UID: \"137de700-2980-41fc-b778-24aa711d2ac2\") " Oct 29 23:45:22.011335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.967866 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137de700-2980-41fc-b778-24aa711d2ac2-kube-api-access-sh5fv\" (UniqueName: \"kubernetes.io/projected/137de700-2980-41fc-b778-24aa711d2ac2-kube-api-access-sh5fv\") pod \"137de700-2980-41fc-b778-24aa711d2ac2\" (UID: \"137de700-2980-41fc-b778-24aa711d2ac2\") " Oct 29 23:45:22.011335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:21.986784 5736 scope.go:122] "RemoveContainer" containerID="79b67cf8f1b06e0375da141a7d4d84cba9648bb3e8a4de633404fa82a58bf587" Oct 29 23:45:22.011335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.002898 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137de700-2980-41fc-b778-24aa711d2ac2-maas-api-service-ca" pod "137de700-2980-41fc-b778-24aa711d2ac2" (UID: "137de700-2980-41fc-b778-24aa711d2ac2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.022820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.022741 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137de700-2980-41fc-b778-24aa711d2ac2-kube-api-access-sh5fv" pod "137de700-2980-41fc-b778-24aa711d2ac2" (UID: "137de700-2980-41fc-b778-24aa711d2ac2"). InnerVolumeSpecName "kube-api-access-sh5fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.112721 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.112681 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-maas-api-service-ca\") pod \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\" (UID: \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\") " Oct 29 23:45:22.112897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.112752 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ff5a60-2de6-41b0-b7c9-0864c9086318-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ff5a60-2de6-41b0-b7c9-0864c9086318-maas-api-service-ca\") pod \"52ff5a60-2de6-41b0-b7c9-0864c9086318\" (UID: \"52ff5a60-2de6-41b0-b7c9-0864c9086318\") " Oct 29 23:45:22.112897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.112824 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-kube-api-access-xnsdc\" (UniqueName: \"kubernetes.io/projected/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-kube-api-access-xnsdc\") pod \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\" (UID: \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\") " Oct 29 23:45:22.112897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.112879 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ff5a60-2de6-41b0-b7c9-0864c9086318-kube-api-access-hxxxn\" (UniqueName: \"kubernetes.io/projected/52ff5a60-2de6-41b0-b7c9-0864c9086318-kube-api-access-hxxxn\") pod \"52ff5a60-2de6-41b0-b7c9-0864c9086318\" (UID: \"52ff5a60-2de6-41b0-b7c9-0864c9086318\") " Oct 29 23:45:22.113150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.113120 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-maas-api-service-ca" pod "4cb9077b-6b07-4a72-907e-b3e9b0e2534a" (UID: "4cb9077b-6b07-4a72-907e-b3e9b0e2534a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.113240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.113134 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-maas-api-service-ca\") pod \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\" (UID: \"bde02fb5-67b1-4da5-9dc3-63bc2b930f08\") " Oct 29 23:45:22.113240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.113159 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ff5a60-2de6-41b0-b7c9-0864c9086318-maas-api-service-ca" pod "52ff5a60-2de6-41b0-b7c9-0864c9086318" (UID: "52ff5a60-2de6-41b0-b7c9-0864c9086318"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.113240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.113231 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a37864a-0480-4fe9-b5c4-c946c1f049a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37864a-0480-4fe9-b5c4-c946c1f049a9-maas-api-service-ca\") pod \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\" (UID: \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\") " Oct 29 23:45:22.114885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.113414 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a37864a-0480-4fe9-b5c4-c946c1f049a9-kube-api-access-c869l\" (UniqueName: \"kubernetes.io/projected/5a37864a-0480-4fe9-b5c4-c946c1f049a9-kube-api-access-c869l\") pod \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\" (UID: \"5a37864a-0480-4fe9-b5c4-c946c1f049a9\") " Oct 29 23:45:22.114885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.113477 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-kube-api-access-8djjl\" (UniqueName: \"kubernetes.io/projected/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-kube-api-access-8djjl\") pod \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\" (UID: \"4cb9077b-6b07-4a72-907e-b3e9b0e2534a\") " Oct 29 23:45:22.114885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.114173 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-maas-api-service-ca" pod "bde02fb5-67b1-4da5-9dc3-63bc2b930f08" (UID: "bde02fb5-67b1-4da5-9dc3-63bc2b930f08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.114885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.114609 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh5fv\" (UniqueName: \"kubernetes.io/projected/137de700-2980-41fc-b778-24aa711d2ac2-kube-api-access-sh5fv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.114885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.114633 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.114885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.114647 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ff5a60-2de6-41b0-b7c9-0864c9086318-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.114885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.114661 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137de700-2980-41fc-b778-24aa711d2ac2-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.114885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.114675 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.114885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.114851 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a37864a-0480-4fe9-b5c4-c946c1f049a9-maas-api-service-ca" pod "5a37864a-0480-4fe9-b5c4-c946c1f049a9" (UID: "5a37864a-0480-4fe9-b5c4-c946c1f049a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.115456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.115430 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ff5a60-2de6-41b0-b7c9-0864c9086318-kube-api-access-hxxxn" pod "52ff5a60-2de6-41b0-b7c9-0864c9086318" (UID: "52ff5a60-2de6-41b0-b7c9-0864c9086318"). InnerVolumeSpecName "kube-api-access-hxxxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.115558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.115455 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-kube-api-access-xnsdc" pod "bde02fb5-67b1-4da5-9dc3-63bc2b930f08" (UID: "bde02fb5-67b1-4da5-9dc3-63bc2b930f08"). InnerVolumeSpecName "kube-api-access-xnsdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.116534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.116487 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-kube-api-access-8djjl" pod "4cb9077b-6b07-4a72-907e-b3e9b0e2534a" (UID: "4cb9077b-6b07-4a72-907e-b3e9b0e2534a"). InnerVolumeSpecName "kube-api-access-8djjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.116981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.116953 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a37864a-0480-4fe9-b5c4-c946c1f049a9-kube-api-access-c869l" pod "5a37864a-0480-4fe9-b5c4-c946c1f049a9" (UID: "5a37864a-0480-4fe9-b5c4-c946c1f049a9"). InnerVolumeSpecName "kube-api-access-c869l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.183126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.183090 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct"] Oct 29 23:45:22.189099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.189070 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b7a567-29888625-6l2ct"] Oct 29 23:45:22.217060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.216962 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c869l\" (UniqueName: \"kubernetes.io/projected/5a37864a-0480-4fe9-b5c4-c946c1f049a9-kube-api-access-c869l\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.217060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.217000 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djjl\" (UniqueName: \"kubernetes.io/projected/4cb9077b-6b07-4a72-907e-b3e9b0e2534a-kube-api-access-8djjl\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.217060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.217017 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnsdc\" (UniqueName: \"kubernetes.io/projected/bde02fb5-67b1-4da5-9dc3-63bc2b930f08-kube-api-access-xnsdc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.217060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.217030 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxxxn\" (UniqueName: \"kubernetes.io/projected/52ff5a60-2de6-41b0-b7c9-0864c9086318-kube-api-access-hxxxn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.217060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.217045 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37864a-0480-4fe9-b5c4-c946c1f049a9-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.222059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.222017 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27"] Oct 29 23:45:22.227136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.227107 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-206095-29888625-c2r27"] Oct 29 23:45:22.262005 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.261965 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk"] Oct 29 23:45:22.268472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.268438 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fb367d-29888625-ff7hk"] Oct 29 23:45:22.275272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.275191 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:45:22.317817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.317778 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc14cbd-79a0-449f-a896-eda15ab761c6-kube-api-access-xggbw\" (UniqueName: \"kubernetes.io/projected/edc14cbd-79a0-449f-a896-eda15ab761c6-kube-api-access-xggbw\") pod \"edc14cbd-79a0-449f-a896-eda15ab761c6\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " Oct 29 23:45:22.318013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.317832 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-tmp-dir\") pod \"edc14cbd-79a0-449f-a896-eda15ab761c6\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " Oct 29 23:45:22.318135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.318112 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-dshm\") pod \"edc14cbd-79a0-449f-a896-eda15ab761c6\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " Oct 29 23:45:22.318286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.318148 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edc14cbd-79a0-449f-a896-eda15ab761c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/edc14cbd-79a0-449f-a896-eda15ab761c6-tls-certs\") pod \"edc14cbd-79a0-449f-a896-eda15ab761c6\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " Oct 29 23:45:22.318286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.318187 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-model-cache\") pod \"edc14cbd-79a0-449f-a896-eda15ab761c6\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " Oct 29 23:45:22.318286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.318190 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-tmp-dir" pod "edc14cbd-79a0-449f-a896-eda15ab761c6" (UID: "edc14cbd-79a0-449f-a896-eda15ab761c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:22.318495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.318290 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-home\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-home\") pod \"edc14cbd-79a0-449f-a896-eda15ab761c6\" (UID: \"edc14cbd-79a0-449f-a896-eda15ab761c6\") " Oct 29 23:45:22.319224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.319197 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.319539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.319517 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-home" pod "edc14cbd-79a0-449f-a896-eda15ab761c6" (UID: "edc14cbd-79a0-449f-a896-eda15ab761c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:22.319996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.319664 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-model-cache" pod "edc14cbd-79a0-449f-a896-eda15ab761c6" (UID: "edc14cbd-79a0-449f-a896-eda15ab761c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:22.321149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.321123 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc14cbd-79a0-449f-a896-eda15ab761c6-tls-certs" pod "edc14cbd-79a0-449f-a896-eda15ab761c6" (UID: "edc14cbd-79a0-449f-a896-eda15ab761c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:22.321284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.321267 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-dshm" pod "edc14cbd-79a0-449f-a896-eda15ab761c6" (UID: "edc14cbd-79a0-449f-a896-eda15ab761c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:22.321351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.321330 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc14cbd-79a0-449f-a896-eda15ab761c6-kube-api-access-xggbw" pod "edc14cbd-79a0-449f-a896-eda15ab761c6" (UID: "edc14cbd-79a0-449f-a896-eda15ab761c6"). InnerVolumeSpecName "kube-api-access-xggbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.421111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.421069 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.421111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.421102 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.421111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.421112 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xggbw\" (UniqueName: \"kubernetes.io/projected/edc14cbd-79a0-449f-a896-eda15ab761c6-kube-api-access-xggbw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.421111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.421121 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc14cbd-79a0-449f-a896-eda15ab761c6-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.421405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.421131 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc14cbd-79a0-449f-a896-eda15ab761c6-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.560756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.560663 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg"] Oct 29 23:45:22.645138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.645100 5736 generic.go:356] "Generic (PLEG): container finished" podID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerID="595f50f203140d89c558326180e6b922b9e880991e601cf8e3fc22911d5bda9d" exitCode=60 Oct 29 23:45:22.645335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.645171 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" event={"ID":"66794c35-2684-4c32-a7c4-7df95beb1b5c","Type":"ContainerDied","Data":"595f50f203140d89c558326180e6b922b9e880991e601cf8e3fc22911d5bda9d"} Oct 29 23:45:22.645335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.645268 5736 scope.go:122] "RemoveContainer" containerID="f92f51093d4bc19624ae5a30f27814b2a4a0ae64f2d32890c50baa8b942525e9" Oct 29 23:45:22.645830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.645808 5736 scope.go:122] "RemoveContainer" containerID="595f50f203140d89c558326180e6b922b9e880991e601cf8e3fc22911d5bda9d" Oct 29 23:45:22.646531 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:22.646247 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5_odh-ai-gateway-infra(66794c35-2684-4c32-a7c4-7df95beb1b5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" Oct 29 23:45:22.646993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.646974 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" Oct 29 23:45:22.647085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.646983 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr" event={"ID":"137de700-2980-41fc-b778-24aa711d2ac2","Type":"ContainerDied","Data":"7b007e27d3a0a36ec18781051c4ebc8d1ae9b8927df80942f83b2e4c2da3b655"} Oct 29 23:45:22.649266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.649230 5736 generic.go:356] "Generic (PLEG): container finished" podID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerID="da39cbfc29bd1ba8121d97e1fe69b35700cfe3aed17f9e5709ef18b0621ac040" exitCode=60 Oct 29 23:45:22.649374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.649315 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" event={"ID":"7055786b-a7d7-4894-81c3-65ac4348fea9","Type":"ContainerDied","Data":"da39cbfc29bd1ba8121d97e1fe69b35700cfe3aed17f9e5709ef18b0621ac040"} Oct 29 23:45:22.653715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.653686 5736 generic.go:356] "Generic (PLEG): container finished" podID="edc14cbd-79a0-449f-a896-eda15ab761c6" containerID="ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171" exitCode=0 Oct 29 23:45:22.653850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.653748 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" Oct 29 23:45:22.653850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.653761 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" event={"ID":"edc14cbd-79a0-449f-a896-eda15ab761c6","Type":"ContainerDied","Data":"ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171"} Oct 29 23:45:22.653850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.653802 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j" event={"ID":"edc14cbd-79a0-449f-a896-eda15ab761c6","Type":"ContainerDied","Data":"438bf51db92c2a4df0394d101be4251d5b828a258e6d522f3a22412bd3303ab1"} Oct 29 23:45:22.658048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.656709 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" Oct 29 23:45:22.658048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.656733 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6" event={"ID":"52ff5a60-2de6-41b0-b7c9-0864c9086318","Type":"ContainerDied","Data":"2e0c481ec16d4044d6ade7a9ec6377d8a2a8498cf82ebbe57e3efc6a6e989a4c"} Oct 29 23:45:22.660628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.659323 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" Oct 29 23:45:22.660628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.659703 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858" event={"ID":"bde02fb5-67b1-4da5-9dc3-63bc2b930f08","Type":"ContainerDied","Data":"872e5316c799b42b21ff3e0470b0110e62c519015d2aad5fb1dc503993e0d141"} Oct 29 23:45:22.666976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.666941 5736 generic.go:356] "Generic (PLEG): container finished" podID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerID="6bce52f5b27674cec7b70b9f0e22ef7af6a20ddc710a23235f672e55d98cc50e" exitCode=60 Oct 29 23:45:22.667082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.667051 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" event={"ID":"1e5b2e09-ffc2-408c-a7a7-97b078c2984e","Type":"ContainerDied","Data":"6bce52f5b27674cec7b70b9f0e22ef7af6a20ddc710a23235f672e55d98cc50e"} Oct 29 23:45:22.667536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.667517 5736 scope.go:122] "RemoveContainer" containerID="6bce52f5b27674cec7b70b9f0e22ef7af6a20ddc710a23235f672e55d98cc50e" Oct 29 23:45:22.668000 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:22.667963 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x_odh-ai-gateway-infra(1e5b2e09-ffc2-408c-a7a7-97b078c2984e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" Oct 29 23:45:22.669187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.669154 5736 scope.go:122] "RemoveContainer" containerID="334283a1c879efc18413746784447ba84e18f7c9b8d10de6bf684af7be34d620" Oct 29 23:45:22.674166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.672889 5736 generic.go:356] "Generic (PLEG): container finished" podID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerID="a0a52544de6389f79aba33a5c71764e9ee85ff0fdfd3e91a12f88fddb592f2df" exitCode=60 Oct 29 23:45:22.674166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.673019 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" event={"ID":"74c99f0d-50d0-4226-b5b7-fd1175bedb56","Type":"ContainerDied","Data":"a0a52544de6389f79aba33a5c71764e9ee85ff0fdfd3e91a12f88fddb592f2df"} Oct 29 23:45:22.674166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.673842 5736 scope.go:122] "RemoveContainer" containerID="a0a52544de6389f79aba33a5c71764e9ee85ff0fdfd3e91a12f88fddb592f2df" Oct 29 23:45:22.674371 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:22.674340 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz_odh-ai-gateway-infra(74c99f0d-50d0-4226-b5b7-fd1175bedb56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" Oct 29 23:45:22.676190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.676165 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" Oct 29 23:45:22.676339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.676166 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr" event={"ID":"5a37864a-0480-4fe9-b5c4-c946c1f049a9","Type":"ContainerDied","Data":"121fca376f2e49eff18841cba9c03f05d46d6dd88f2da00d8c05e6fabe41c375"} Oct 29 23:45:22.679335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.679295 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" Oct 29 23:45:22.679724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.679694 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8" event={"ID":"4cb9077b-6b07-4a72-907e-b3e9b0e2534a","Type":"ContainerDied","Data":"cd9477cc65910e0d8caee1629377d472b1cfe27b7751b5cda2cf06e0a418f801"} Oct 29 23:45:22.691013 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.689773 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5"] Oct 29 23:45:22.692798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.692419 5736 generic.go:356] "Generic (PLEG): container finished" podID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerID="8091bcf6056ee5a19af154cf5dfb34b960f947aa44ec5015d6b963b460d28927" exitCode=60 Oct 29 23:45:22.692798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.692462 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" event={"ID":"bd3eb218-be48-4f91-b4ca-94675bca35f5","Type":"ContainerDied","Data":"8091bcf6056ee5a19af154cf5dfb34b960f947aa44ec5015d6b963b460d28927"} Oct 29 23:45:22.694124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.693648 5736 scope.go:122] "RemoveContainer" containerID="8091bcf6056ee5a19af154cf5dfb34b960f947aa44ec5015d6b963b460d28927" Oct 29 23:45:22.694396 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:22.694324 5736 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4_odh-ai-gateway-infra(bd3eb218-be48-4f91-b4ca-94675bca35f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" Oct 29 23:45:22.701837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.699981 5736 scope.go:122] "RemoveContainer" containerID="03d5d714d8fab3c0c20a154384fcb006372bbedf7fc8fd50c252cc1585ddae0b" Oct 29 23:45:22.731923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.731891 5736 scope.go:122] "RemoveContainer" containerID="ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171" Oct 29 23:45:22.748023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.747977 5736 scope.go:122] "RemoveContainer" containerID="ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171" Oct 29 23:45:22.748735 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:22.748706 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171\": container with ID starting with ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171 not found: ID does not exist" containerID="ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171" Oct 29 23:45:22.748918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.748888 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171"} err="failed to get container status \"ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171\": rpc error: code = NotFound desc = could not find container \"ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171\": container with ID starting with ab2fc8e9d8b770042dbbea99f764a8e42f8557283bdc0ae6a76fa98d99cd6171 not found: ID does not exist" Oct 29 23:45:22.749057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.749042 5736 scope.go:122] "RemoveContainer" containerID="3970be43072cf62fbfe60e5ab75d4b63cbfb699b769e7912ef873f6435dffb42" Oct 29 23:45:22.771698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.763695 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6"] Oct 29 23:45:22.771698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.770413 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4ad17e-29888625-7gkv6"] Oct 29 23:45:22.772031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.771876 5736 scope.go:122] "RemoveContainer" containerID="a3ec2dbc07510300b5c81702e18fb29172c585932776a353ef98a27d99d0db3c" Oct 29 23:45:22.803868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.803838 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4"] Oct 29 23:45:22.804061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.804028 5736 scope.go:122] "RemoveContainer" containerID="e9e2dcc7d16d32e53218391658927434472eaf418e60e702b4d08d3dc2cd16d2" Oct 29 23:45:22.810235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.810208 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr"] Oct 29 23:45:22.821535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.821459 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-750972-29888625-q7hvr"] Oct 29 23:45:22.827302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.827282 5736 scope.go:122] "RemoveContainer" containerID="51c0e40b09398c2c99e4b1f1b64e2dd1abad50b5dcf6e53c5a9f4637ea735b59" Oct 29 23:45:22.831151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.831125 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x"] Oct 29 23:45:22.866309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.866280 5736 scope.go:122] "RemoveContainer" containerID="939011f0ca24fa8d3885fa6679507176af9a62a5894e7576b2bb0affd694c010" Oct 29 23:45:22.904699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.904657 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j"] Oct 29 23:45:22.907239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.907214 5736 scope.go:122] "RemoveContainer" containerID="20738a2be0d2525d945e67464f9e3d2a22a990ca194d039a18a73b1ff2d9b8a9" Oct 29 23:45:22.909712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.909685 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b7a567/test-model-b7a567-kserve-7b8966f899-w5z4j"] Oct 29 23:45:22.911514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.911480 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:22.933008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.932841 5736 scope.go:122] "RemoveContainer" containerID="124019b61304223e01bd7a0ac6f8824ea3911111cb02f99dc37a5bf6285e0f4e" Oct 29 23:45:22.946856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.946283 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr"] Oct 29 23:45:22.949147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.949116 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7055786b-a7d7-4894-81c3-65ac4348fea9-kube-api-access-8jmcc\" (UniqueName: \"kubernetes.io/projected/7055786b-a7d7-4894-81c3-65ac4348fea9-kube-api-access-8jmcc\") pod \"7055786b-a7d7-4894-81c3-65ac4348fea9\" (UID: \"7055786b-a7d7-4894-81c3-65ac4348fea9\") " Oct 29 23:45:22.951792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.951760 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7055786b-a7d7-4894-81c3-65ac4348fea9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7055786b-a7d7-4894-81c3-65ac4348fea9-maas-api-service-ca\") pod \"7055786b-a7d7-4894-81c3-65ac4348fea9\" (UID: \"7055786b-a7d7-4894-81c3-65ac4348fea9\") " Oct 29 23:45:22.952987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.952239 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7055786b-a7d7-4894-81c3-65ac4348fea9-maas-api-service-ca" pod "7055786b-a7d7-4894-81c3-65ac4348fea9" (UID: "7055786b-a7d7-4894-81c3-65ac4348fea9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:22.952987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.952943 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7055786b-a7d7-4894-81c3-65ac4348fea9-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:22.953901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.953864 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7055786b-a7d7-4894-81c3-65ac4348fea9-kube-api-access-8jmcc" pod "7055786b-a7d7-4894-81c3-65ac4348fea9" (UID: "7055786b-a7d7-4894-81c3-65ac4348fea9"). InnerVolumeSpecName "kube-api-access-8jmcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:22.965804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.965776 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:22.967511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.967482 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3b8149-29888625-56fgr"] Oct 29 23:45:22.975773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:22.975742 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz"] Oct 29 23:45:23.009779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.009743 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137de700-2980-41fc-b778-24aa711d2ac2" path="/var/lib/kubelet/pods/137de700-2980-41fc-b778-24aa711d2ac2/volumes" Oct 29 23:45:23.010470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.010445 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" path="/var/lib/kubelet/pods/52ff5a60-2de6-41b0-b7c9-0864c9086318/volumes" Oct 29 23:45:23.011045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.011015 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" path="/var/lib/kubelet/pods/5a37864a-0480-4fe9-b5c4-c946c1f049a9/volumes" Oct 29 23:45:23.012124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.012098 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" path="/var/lib/kubelet/pods/aacb0496-adc4-4859-afe2-88a8c07d27b3/volumes" Oct 29 23:45:23.012661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.012637 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" path="/var/lib/kubelet/pods/b61d2aa1-55ea-46a6-adbd-f7a03cf883fb/volumes" Oct 29 23:45:23.013256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.013234 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" path="/var/lib/kubelet/pods/bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa/volumes" Oct 29 23:45:23.014303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.014271 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" path="/var/lib/kubelet/pods/dd63cd40-3af1-49cb-b17b-5576a1f434cd/volumes" Oct 29 23:45:23.014916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.014894 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc14cbd-79a0-449f-a896-eda15ab761c6" path="/var/lib/kubelet/pods/edc14cbd-79a0-449f-a896-eda15ab761c6/volumes" Oct 29 23:45:23.015383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.015361 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" path="/var/lib/kubelet/pods/fefefc63-6416-4fec-b277-6a8b9e9de6fc/volumes" Oct 29 23:45:23.016365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.016337 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:23.032065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.032036 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8"] Oct 29 23:45:23.038911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.038881 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2591f6-29888625-tkpl8"] Oct 29 23:45:23.054220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.054189 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e36a42-a6d4-449b-8923-f3a061fd58ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e36a42-a6d4-449b-8923-f3a061fd58ec-maas-api-service-ca\") pod \"80e36a42-a6d4-449b-8923-f3a061fd58ec\" (UID: \"80e36a42-a6d4-449b-8923-f3a061fd58ec\") " Oct 29 23:45:23.054861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.054741 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e36a42-a6d4-449b-8923-f3a061fd58ec-maas-api-service-ca" pod "80e36a42-a6d4-449b-8923-f3a061fd58ec" (UID: "80e36a42-a6d4-449b-8923-f3a061fd58ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:23.055162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.055141 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e36a42-a6d4-449b-8923-f3a061fd58ec-kube-api-access-kfd98\" (UniqueName: \"kubernetes.io/projected/80e36a42-a6d4-449b-8923-f3a061fd58ec-kube-api-access-kfd98\") pod \"80e36a42-a6d4-449b-8923-f3a061fd58ec\" (UID: \"80e36a42-a6d4-449b-8923-f3a061fd58ec\") " Oct 29 23:45:23.055761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.055740 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jmcc\" (UniqueName: \"kubernetes.io/projected/7055786b-a7d7-4894-81c3-65ac4348fea9-kube-api-access-8jmcc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.055846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.055764 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e36a42-a6d4-449b-8923-f3a061fd58ec-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.057417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.057394 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e36a42-a6d4-449b-8923-f3a061fd58ec-kube-api-access-kfd98" pod "80e36a42-a6d4-449b-8923-f3a061fd58ec" (UID: "80e36a42-a6d4-449b-8923-f3a061fd58ec"). InnerVolumeSpecName "kube-api-access-kfd98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:23.156257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.156226 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858"] Oct 29 23:45:23.156860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.156837 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c9e79f9-8c22-4a9f-87db-bd6413409296-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e79f9-8c22-4a9f-87db-bd6413409296-maas-api-service-ca\") pod \"8c9e79f9-8c22-4a9f-87db-bd6413409296\" (UID: \"8c9e79f9-8c22-4a9f-87db-bd6413409296\") " Oct 29 23:45:23.157010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.156993 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9e79f9-8c22-4a9f-87db-bd6413409296-kube-api-access-vffw9\" (UniqueName: \"kubernetes.io/projected/8c9e79f9-8c22-4a9f-87db-bd6413409296-kube-api-access-vffw9\") pod \"8c9e79f9-8c22-4a9f-87db-bd6413409296\" (UID: \"8c9e79f9-8c22-4a9f-87db-bd6413409296\") " Oct 29 23:45:23.157250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.157224 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9e79f9-8c22-4a9f-87db-bd6413409296-maas-api-service-ca" pod "8c9e79f9-8c22-4a9f-87db-bd6413409296" (UID: "8c9e79f9-8c22-4a9f-87db-bd6413409296"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:23.157813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.157793 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e79f9-8c22-4a9f-87db-bd6413409296-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.157900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.157819 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfd98\" (UniqueName: \"kubernetes.io/projected/80e36a42-a6d4-449b-8923-f3a061fd58ec-kube-api-access-kfd98\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.159351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.159319 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9e79f9-8c22-4a9f-87db-bd6413409296-kube-api-access-vffw9" pod "8c9e79f9-8c22-4a9f-87db-bd6413409296" (UID: "8c9e79f9-8c22-4a9f-87db-bd6413409296"). InnerVolumeSpecName "kube-api-access-vffw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:23.163742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.163716 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a84df8-29888625-pq858"] Oct 29 23:45:23.261643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.261551 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vffw9\" (UniqueName: \"kubernetes.io/projected/8c9e79f9-8c22-4a9f-87db-bd6413409296-kube-api-access-vffw9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:23.618269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.618219 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" podUID="88078911-4efa-4a76-82d1-70bce5369a7c" containerName="payload-processing" containerID="cri-o://adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226" gracePeriod=25 Oct 29 23:45:23.656842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.656796 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" podUID="d640b153-10af-4412-8334-97e3f60cb32f" containerName="payload-pre-processing" containerID="cri-o://afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c" gracePeriod=25 Oct 29 23:45:23.707931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.707901 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" Oct 29 23:45:23.708102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.707894 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5" event={"ID":"80e36a42-a6d4-449b-8923-f3a061fd58ec","Type":"ContainerDied","Data":"23ef8f3573c97461cc8b3f9d3a501d9e9ce7681f9575d0c2797a14ca4c73ea87"} Oct 29 23:45:23.708163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.708099 5736 scope.go:122] "RemoveContainer" containerID="7c4217ebf5501ae3d45daef17c55ca479439654298025a4efd16f82b23dbab77" Oct 29 23:45:23.716737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.716709 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" Oct 29 23:45:23.717031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.717001 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4" event={"ID":"8c9e79f9-8c22-4a9f-87db-bd6413409296","Type":"ContainerDied","Data":"cc3d9953fe86bc572288aa9f8cd0042d3b2d5d0f94ea5ba264909b6640bd1e59"} Oct 29 23:45:23.725095 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.724721 5736 scope.go:122] "RemoveContainer" containerID="b3641258aac00f10b965560bf59a74bfdb9242972a6ec0c8393b04025e0d4436" Oct 29 23:45:23.729504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.729426 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" event={"ID":"7055786b-a7d7-4894-81c3-65ac4348fea9","Type":"ContainerDied","Data":"5ff398b0bc663f88362ad605c77ad14de564d70f1c47e86c750d3e929a067da2"} Oct 29 23:45:23.729803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.729641 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg" Oct 29 23:45:23.747868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.747838 5736 scope.go:122] "RemoveContainer" containerID="da39cbfc29bd1ba8121d97e1fe69b35700cfe3aed17f9e5709ef18b0621ac040" Oct 29 23:45:23.898370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.898338 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5"] Oct 29 23:45:23.904899 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:23.904869 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dea9cb-29888625-s2zf5"] Oct 29 23:45:24.014810 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.014693 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:24.053631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.048853 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:24.075849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.075812 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c99f0d-50d0-4226-b5b7-fd1175bedb56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c99f0d-50d0-4226-b5b7-fd1175bedb56-maas-api-service-ca\") pod \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\" (UID: \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\") " Oct 29 23:45:24.076023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.075884 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c99f0d-50d0-4226-b5b7-fd1175bedb56-kube-api-access-97knw\" (UniqueName: \"kubernetes.io/projected/74c99f0d-50d0-4226-b5b7-fd1175bedb56-kube-api-access-97knw\") pod \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\" (UID: \"74c99f0d-50d0-4226-b5b7-fd1175bedb56\") " Oct 29 23:45:24.076023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.075918 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-maas-api-service-ca\") pod \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\" (UID: \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\") " Oct 29 23:45:24.076220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.076195 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-kube-api-access-bqtwv\" (UniqueName: \"kubernetes.io/projected/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-kube-api-access-bqtwv\") pod \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\" (UID: \"1e5b2e09-ffc2-408c-a7a7-97b078c2984e\") " Oct 29 23:45:24.076318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.076221 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c99f0d-50d0-4226-b5b7-fd1175bedb56-maas-api-service-ca" pod "74c99f0d-50d0-4226-b5b7-fd1175bedb56" (UID: "74c99f0d-50d0-4226-b5b7-fd1175bedb56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:24.077168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.077143 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c99f0d-50d0-4226-b5b7-fd1175bedb56-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:24.079786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.079752 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-kube-api-access-bqtwv" pod "1e5b2e09-ffc2-408c-a7a7-97b078c2984e" (UID: "1e5b2e09-ffc2-408c-a7a7-97b078c2984e"). InnerVolumeSpecName "kube-api-access-bqtwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:24.080866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.076727 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-maas-api-service-ca" pod "1e5b2e09-ffc2-408c-a7a7-97b078c2984e" (UID: "1e5b2e09-ffc2-408c-a7a7-97b078c2984e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:24.081479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.081429 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c99f0d-50d0-4226-b5b7-fd1175bedb56-kube-api-access-97knw" pod "74c99f0d-50d0-4226-b5b7-fd1175bedb56" (UID: "74c99f0d-50d0-4226-b5b7-fd1175bedb56"). InnerVolumeSpecName "kube-api-access-97knw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:24.109252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.109220 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg"] Oct 29 23:45:24.116472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.116437 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51462d-29888625-psflg"] Oct 29 23:45:24.117160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.117134 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:24.120542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.120514 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:24.145606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.145375 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4"] Oct 29 23:45:24.149431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.149404 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d20345-29888625-xxnk4"] Oct 29 23:45:24.178148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.178117 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66794c35-2684-4c32-a7c4-7df95beb1b5c-kube-api-access-zskqf\" (UniqueName: \"kubernetes.io/projected/66794c35-2684-4c32-a7c4-7df95beb1b5c-kube-api-access-zskqf\") pod \"66794c35-2684-4c32-a7c4-7df95beb1b5c\" (UID: \"66794c35-2684-4c32-a7c4-7df95beb1b5c\") " Oct 29 23:45:24.178289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.178235 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3eb218-be48-4f91-b4ca-94675bca35f5-kube-api-access-dsx5p\" (UniqueName: \"kubernetes.io/projected/bd3eb218-be48-4f91-b4ca-94675bca35f5-kube-api-access-dsx5p\") pod \"bd3eb218-be48-4f91-b4ca-94675bca35f5\" (UID: \"bd3eb218-be48-4f91-b4ca-94675bca35f5\") " Oct 29 23:45:24.178341 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.178330 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66794c35-2684-4c32-a7c4-7df95beb1b5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66794c35-2684-4c32-a7c4-7df95beb1b5c-maas-api-service-ca\") pod \"66794c35-2684-4c32-a7c4-7df95beb1b5c\" (UID: \"66794c35-2684-4c32-a7c4-7df95beb1b5c\") " Oct 29 23:45:24.178385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.178362 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3eb218-be48-4f91-b4ca-94675bca35f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3eb218-be48-4f91-b4ca-94675bca35f5-maas-api-service-ca\") pod \"bd3eb218-be48-4f91-b4ca-94675bca35f5\" (UID: \"bd3eb218-be48-4f91-b4ca-94675bca35f5\") " Oct 29 23:45:24.178732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.178709 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66794c35-2684-4c32-a7c4-7df95beb1b5c-maas-api-service-ca" pod "66794c35-2684-4c32-a7c4-7df95beb1b5c" (UID: "66794c35-2684-4c32-a7c4-7df95beb1b5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:24.178853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.178816 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3eb218-be48-4f91-b4ca-94675bca35f5-maas-api-service-ca" pod "bd3eb218-be48-4f91-b4ca-94675bca35f5" (UID: "bd3eb218-be48-4f91-b4ca-94675bca35f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:24.179073 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.179053 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqtwv\" (UniqueName: \"kubernetes.io/projected/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-kube-api-access-bqtwv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:24.179169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.179074 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97knw\" (UniqueName: \"kubernetes.io/projected/74c99f0d-50d0-4226-b5b7-fd1175bedb56-kube-api-access-97knw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:24.179169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.179085 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5b2e09-ffc2-408c-a7a7-97b078c2984e-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:24.179169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.179093 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66794c35-2684-4c32-a7c4-7df95beb1b5c-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:24.179169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.179103 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3eb218-be48-4f91-b4ca-94675bca35f5-maas-api-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:24.180498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.180473 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3eb218-be48-4f91-b4ca-94675bca35f5-kube-api-access-dsx5p" pod "bd3eb218-be48-4f91-b4ca-94675bca35f5" (UID: "bd3eb218-be48-4f91-b4ca-94675bca35f5"). InnerVolumeSpecName "kube-api-access-dsx5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:24.180599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.180512 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66794c35-2684-4c32-a7c4-7df95beb1b5c-kube-api-access-zskqf" pod "66794c35-2684-4c32-a7c4-7df95beb1b5c" (UID: "66794c35-2684-4c32-a7c4-7df95beb1b5c"). InnerVolumeSpecName "kube-api-access-zskqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:24.281209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.281157 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsx5p\" (UniqueName: \"kubernetes.io/projected/bd3eb218-be48-4f91-b4ca-94675bca35f5-kube-api-access-dsx5p\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:24.281209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.281209 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zskqf\" (UniqueName: \"kubernetes.io/projected/66794c35-2684-4c32-a7c4-7df95beb1b5c-kube-api-access-zskqf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:24.742195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.742161 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" Oct 29 23:45:24.742733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.742199 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4" event={"ID":"bd3eb218-be48-4f91-b4ca-94675bca35f5","Type":"ContainerDied","Data":"80d877966ed1b8d00d290d97906e7875a507ed9c2d1500771271e4ce9c0c2774"} Oct 29 23:45:24.742733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.742281 5736 scope.go:122] "RemoveContainer" containerID="8091bcf6056ee5a19af154cf5dfb34b960f947aa44ec5015d6b963b460d28927" Oct 29 23:45:24.744042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.744019 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" Oct 29 23:45:24.744042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.744032 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5" event={"ID":"66794c35-2684-4c32-a7c4-7df95beb1b5c","Type":"ContainerDied","Data":"24c0af018f0633b5981ee139012fcf25bf3860cb88411cd33ec612f46016c9d3"} Oct 29 23:45:24.748358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.748327 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" Oct 29 23:45:24.748454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.748353 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x" event={"ID":"1e5b2e09-ffc2-408c-a7a7-97b078c2984e","Type":"ContainerDied","Data":"087f4f9169b6749c162f173f8764c2d95790e9003f2fc8a884f5950e87ab6c9e"} Oct 29 23:45:24.769336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.769309 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" Oct 29 23:45:24.769449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.769304 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz" event={"ID":"74c99f0d-50d0-4226-b5b7-fd1175bedb56","Type":"ContainerDied","Data":"09ec3a273928be7137e0779b1031df4bf2db098a6d395c648b7eff6b301ed663"} Oct 29 23:45:24.777282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.777259 5736 scope.go:122] "RemoveContainer" containerID="595f50f203140d89c558326180e6b922b9e880991e601cf8e3fc22911d5bda9d" Oct 29 23:45:24.788133 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.788108 5736 scope.go:122] "RemoveContainer" containerID="6bce52f5b27674cec7b70b9f0e22ef7af6a20ddc710a23235f672e55d98cc50e" Oct 29 23:45:24.803554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.803527 5736 scope.go:122] "RemoveContainer" containerID="a0a52544de6389f79aba33a5c71764e9ee85ff0fdfd3e91a12f88fddb592f2df" Oct 29 23:45:24.816926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.816902 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4"] Oct 29 23:45:24.824468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.824442 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8a8b61-29888625-6zkt4"] Oct 29 23:45:24.922435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.922400 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x"] Oct 29 23:45:24.948568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.948537 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e1769a-29888625-spl4x"] Oct 29 23:45:24.990026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.989990 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz"] Oct 29 23:45:24.994343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:24.994282 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c3fc-29888625-5xppz"] Oct 29 23:45:25.006220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.006190 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" path="/var/lib/kubelet/pods/1e5b2e09-ffc2-408c-a7a7-97b078c2984e/volumes" Oct 29 23:45:25.006712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.006697 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" path="/var/lib/kubelet/pods/4cb9077b-6b07-4a72-907e-b3e9b0e2534a/volumes" Oct 29 23:45:25.007152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.007136 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" path="/var/lib/kubelet/pods/7055786b-a7d7-4894-81c3-65ac4348fea9/volumes" Oct 29 23:45:25.007880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.007864 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" path="/var/lib/kubelet/pods/74c99f0d-50d0-4226-b5b7-fd1175bedb56/volumes" Oct 29 23:45:25.008304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.008290 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" path="/var/lib/kubelet/pods/80e36a42-a6d4-449b-8923-f3a061fd58ec/volumes" Oct 29 23:45:25.008727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.008713 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" path="/var/lib/kubelet/pods/8c9e79f9-8c22-4a9f-87db-bd6413409296/volumes" Oct 29 23:45:25.009446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.009423 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" path="/var/lib/kubelet/pods/bd3eb218-be48-4f91-b4ca-94675bca35f5/volumes" Oct 29 23:45:25.009949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.009927 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" path="/var/lib/kubelet/pods/bde02fb5-67b1-4da5-9dc3-63bc2b930f08/volumes" Oct 29 23:45:25.027395 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.027369 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5"] Oct 29 23:45:25.031184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.031161 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b4b5e3-29888625-6m2j5"] Oct 29 23:45:25.531639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.520593 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz"] Oct 29 23:45:25.531639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:25.520844 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" containerID="cri-o://094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15" gracePeriod=30 Oct 29 23:45:26.209974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.209941 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s"] Oct 29 23:45:26.210747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210731 5736 cpu_manager.go:414] "containerMap: removing container" podUID="137de700-2980-41fc-b778-24aa711d2ac2" containerName="cleanup" Oct 29 23:45:26.210814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210748 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137de700-2980-41fc-b778-24aa711d2ac2" containerName="cleanup" Oct 29 23:45:26.210814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210764 5736 cpu_manager.go:414] "containerMap: removing container" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerName="cleanup" Oct 29 23:45:26.210814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210771 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerName="cleanup" Oct 29 23:45:26.210814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210783 5736 cpu_manager.go:414] "containerMap: removing container" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerName="cleanup" Oct 29 23:45:26.210814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210787 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerName="cleanup" Oct 29 23:45:26.210814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210794 5736 cpu_manager.go:414] "containerMap: removing container" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerName="cleanup" Oct 29 23:45:26.210814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210799 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerName="cleanup" Oct 29 23:45:26.210814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210806 5736 cpu_manager.go:414] "containerMap: removing container" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerName="cleanup" Oct 29 23:45:26.210814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210811 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210818 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9b94d6d5-7d12-4c1e-9a2d-060b12d6e338" containerName="revoke-keys" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210823 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b94d6d5-7d12-4c1e-9a2d-060b12d6e338" containerName="revoke-keys" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210833 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210838 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210846 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210850 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210857 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210862 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210874 5736 cpu_manager.go:414] "containerMap: removing container" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210879 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210888 5736 cpu_manager.go:414] "containerMap: removing container" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210892 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210899 5736 cpu_manager.go:414] "containerMap: removing container" podUID="edc14cbd-79a0-449f-a896-eda15ab761c6" containerName="main" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210903 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc14cbd-79a0-449f-a896-eda15ab761c6" containerName="main" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210912 5736 cpu_manager.go:414] "containerMap: removing container" podUID="137de700-2980-41fc-b778-24aa711d2ac2" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210916 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137de700-2980-41fc-b778-24aa711d2ac2" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210923 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210927 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210933 5736 cpu_manager.go:414] "containerMap: removing container" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210938 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210944 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210949 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210956 5736 cpu_manager.go:414] "containerMap: removing container" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210960 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210967 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210972 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210978 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210982 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210991 5736 cpu_manager.go:414] "containerMap: removing container" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.210996 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211003 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211007 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211014 5736 cpu_manager.go:414] "containerMap: removing container" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211018 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211024 5736 cpu_manager.go:414] "containerMap: removing container" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211028 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211046 5736 cpu_manager.go:414] "containerMap: removing container" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211050 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211057 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211062 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211070 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211075 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211081 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211086 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211093 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211098 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211108 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211113 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211118 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211123 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211132 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211137 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211145 5736 cpu_manager.go:414] "containerMap: removing container" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211149 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211155 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211160 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211167 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211171 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211178 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerName="cleanup" Oct 29 23:45:26.211155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211182 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211190 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211194 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211201 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211206 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211216 5736 cpu_manager.go:414] "containerMap: removing container" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211220 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211232 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211236 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211242 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211246 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211256 5736 cpu_manager.go:414] "containerMap: removing container" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211261 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211269 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211273 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211281 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211285 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211301 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211305 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211412 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211422 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211432 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211442 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211453 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211462 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211467 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211474 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211480 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211487 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="137de700-2980-41fc-b778-24aa711d2ac2" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211492 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211499 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211504 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211510 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211516 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211522 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211528 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211533 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3eb218-be48-4f91-b4ca-94675bca35f5" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211541 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211546 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5b2e09-ffc2-408c-a7a7-97b078c2984e" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211553 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211558 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211564 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211570 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211589 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211600 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211608 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211618 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211626 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211631 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7055786b-a7d7-4894-81c3-65ac4348fea9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211639 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211644 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211650 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211658 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211665 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211671 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211676 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b94d6d5-7d12-4c1e-9a2d-060b12d6e338" containerName="revoke-keys" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211684 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211691 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211699 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc14cbd-79a0-449f-a896-eda15ab761c6" containerName="main" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211705 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="137de700-2980-41fc-b778-24aa711d2ac2" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211712 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211716 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211723 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerName="cleanup" Oct 29 23:45:26.213049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.211730 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerName="cleanup" Oct 29 23:45:26.217640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.217624 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.235495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.235468 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s"] Oct 29 23:45:26.323531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.323497 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:45:26.323531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.323501 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:45:26.337531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.337498 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.337762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.337741 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5md5\" (UniqueName: \"kubernetes.io/projected/1b5939aa-428e-49c3-bdd7-4982bb610cf1-kube-api-access-d5md5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.337920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.337905 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.439736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.439696 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.439937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.439800 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.440038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.440021 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5md5\" (UniqueName: \"kubernetes.io/projected/1b5939aa-428e-49c3-bdd7-4982bb610cf1-kube-api-access-d5md5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.440365 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.440342 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.442231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.442204 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.463009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.462947 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5md5\" (UniqueName: \"kubernetes.io/projected/1b5939aa-428e-49c3-bdd7-4982bb610cf1-kube-api-access-d5md5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.532597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.532552 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:26.826801 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:26.826720 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:45:26.866598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.865828 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s"] Oct 29 23:45:26.870199 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:26.870166 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5939aa_428e_49c3_bdd7_4982bb610cf1.slice/crio-4ff2d27bb52857b90b8154a4d055f446c99c9601ee56210240ebce2505925871 WatchSource:0}: Error finding container 4ff2d27bb52857b90b8154a4d055f446c99c9601ee56210240ebce2505925871: Status 404 returned error can't find the container with id 4ff2d27bb52857b90b8154a4d055f446c99c9601ee56210240ebce2505925871 Oct 29 23:45:26.962772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:26.962729 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" event={"ID":"1b5939aa-428e-49c3-bdd7-4982bb610cf1","Type":"ContainerStarted","Data":"4ff2d27bb52857b90b8154a4d055f446c99c9601ee56210240ebce2505925871"} Oct 29 23:45:27.015165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:27.015123 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" path="/var/lib/kubelet/pods/66794c35-2684-4c32-a7c4-7df95beb1b5c/volumes" Oct 29 23:45:27.999619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:27.999535 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" event={"ID":"1b5939aa-428e-49c3-bdd7-4982bb610cf1","Type":"ContainerStarted","Data":"bcff04e056bebeaa66a8682240560a9a942efd9b450bd068158dd8f0793e3b51"} Oct 29 23:45:28.043053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.042993 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" podStartSLOduration=2.042977495 podStartE2EDuration="2.042977495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:28.03880443 +0000 UTC m=+3725.154724037" watchObservedRunningTime="2026-10-29 23:45:28.042977495 +0000 UTC m=+3725.158897101" Oct 29 23:45:28.761745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.761707 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w"] Oct 29 23:45:28.762840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762819 5736 cpu_manager.go:414] "containerMap: removing container" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerName="cleanup" Oct 29 23:45:28.762935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762842 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerName="cleanup" Oct 29 23:45:28.762935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762869 5736 cpu_manager.go:414] "containerMap: removing container" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerName="cleanup" Oct 29 23:45:28.762935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762877 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ff5a60-2de6-41b0-b7c9-0864c9086318" containerName="cleanup" Oct 29 23:45:28.762935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762899 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerName="cleanup" Oct 29 23:45:28.762935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762907 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerName="cleanup" Oct 29 23:45:28.762935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762929 5736 cpu_manager.go:414] "containerMap: removing container" podUID="137de700-2980-41fc-b778-24aa711d2ac2" containerName="cleanup" Oct 29 23:45:28.762935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762936 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137de700-2980-41fc-b778-24aa711d2ac2" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762949 5736 cpu_manager.go:414] "containerMap: removing container" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762956 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db376677-f21f-4807-ab75-b82ce5e8f51b" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762968 5736 cpu_manager.go:414] "containerMap: removing container" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762975 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.762993 5736 cpu_manager.go:414] "containerMap: removing container" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763001 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763013 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763020 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763036 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763046 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763060 5736 cpu_manager.go:414] "containerMap: removing container" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763067 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd63cd40-3af1-49cb-b17b-5576a1f434cd" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763080 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763087 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763102 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763109 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb9077b-6b07-4a72-907e-b3e9b0e2534a" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763297 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c99f0d-50d0-4226-b5b7-fd1175bedb56" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763311 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefefc63-6416-4fec-b277-6a8b9e9de6fc" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763326 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a37864a-0480-4fe9-b5c4-c946c1f049a9" containerName="cleanup" Oct 29 23:45:28.763332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763340 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="66794c35-2684-4c32-a7c4-7df95beb1b5c" containerName="cleanup" Oct 29 23:45:28.764695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763351 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacb0496-adc4-4859-afe2-88a8c07d27b3" containerName="cleanup" Oct 29 23:45:28.764695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763363 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e36a42-a6d4-449b-8923-f3a061fd58ec" containerName="cleanup" Oct 29 23:45:28.764695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763373 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1cc52d-92bf-4fa5-afb3-3e4b433f12aa" containerName="cleanup" Oct 29 23:45:28.764695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763384 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="137de700-2980-41fc-b778-24aa711d2ac2" containerName="cleanup" Oct 29 23:45:28.764695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763395 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61d2aa1-55ea-46a6-adbd-f7a03cf883fb" containerName="cleanup" Oct 29 23:45:28.764695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763411 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9e79f9-8c22-4a9f-87db-bd6413409296" containerName="cleanup" Oct 29 23:45:28.764695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.763423 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde02fb5-67b1-4da5-9dc3-63bc2b930f08" containerName="cleanup" Oct 29 23:45:28.769851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.769827 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:28.813868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.804012 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w"] Oct 29 23:45:28.863493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.863462 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-825472-openshift-default-dockercfg-vh6rb\"" Oct 29 23:45:28.908779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.908749 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:28.908916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.908797 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:28.908990 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.908927 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:28.909048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.908988 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:28.909141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.909121 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:28.909211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.909165 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2l2\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-kube-api-access-xd2l2\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:28.909283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.909250 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:28.909364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.909350 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:28.909477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:28.909464 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.010575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.010540 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.011059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.010607 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.011059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.010667 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.011059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.010715 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.011059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.010796 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.011059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.010828 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2l2\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-kube-api-access-xd2l2\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.011059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.010864 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.011059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.010951 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.011455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.011094 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.013423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.011904 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.013423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.011940 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.013423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.012246 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.013423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.012756 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.013423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.013234 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.014858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.014830 5736 generic.go:356] "Generic (PLEG): container finished" podID="1b5939aa-428e-49c3-bdd7-4982bb610cf1" containerID="bcff04e056bebeaa66a8682240560a9a942efd9b450bd068158dd8f0793e3b51" exitCode=0 Oct 29 23:45:29.015143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.014969 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.015263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.015011 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" event={"ID":"1b5939aa-428e-49c3-bdd7-4982bb610cf1","Type":"ContainerDied","Data":"bcff04e056bebeaa66a8682240560a9a942efd9b450bd068158dd8f0793e3b51"} Oct 29 23:45:29.015263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.015215 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.036166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.036136 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.036773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.036754 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2l2\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-kube-api-access-xd2l2\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.086049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.086015 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:29.195497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.195459 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr"] Oct 29 23:45:29.195773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.195740 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" podUID="10608824-0cc9-41b0-b4a9-a8230d797de3" containerName="maas-api" containerID="cri-o://a879deccc3f4d296ac49fb7ad77ad42beda41bf56866fb63c4bde7ec6a76eeb5" gracePeriod=30 Oct 29 23:45:29.478771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.462036 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w"] Oct 29 23:45:29.659929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.605520 5736 scope.go:122] "RemoveContainer" containerID="807b0396f65a3d2036bf77f044ea969c37a8ca879105663e1558e49e787217c7" Oct 29 23:45:29.753603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.748797 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:45:29.753603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.748919 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:45:29.753603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:29.748962 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:45:30.092097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.091833 5736 scope.go:122] "RemoveContainer" containerID="4a55adeae65847c678366790ca80235cf5366402cb2b5dafdee3647ef2ec14c5" Oct 29 23:45:30.097151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.097044 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" event={"ID":"ff57e2f5-af27-42fd-a5ed-0bd386e72344","Type":"ContainerStarted","Data":"30595d17ebb53b51badef740ed1cc4e3ffb75760c98ac13caedee04236678d0f"} Oct 29 23:45:30.100446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.100415 5736 generic.go:356] "Generic (PLEG): container finished" podID="10608824-0cc9-41b0-b4a9-a8230d797de3" containerID="a879deccc3f4d296ac49fb7ad77ad42beda41bf56866fb63c4bde7ec6a76eeb5" exitCode=0 Oct 29 23:45:30.102242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.102213 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" event={"ID":"10608824-0cc9-41b0-b4a9-a8230d797de3","Type":"ContainerDied","Data":"a879deccc3f4d296ac49fb7ad77ad42beda41bf56866fb63c4bde7ec6a76eeb5"} Oct 29 23:45:30.102441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.102421 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" event={"ID":"10608824-0cc9-41b0-b4a9-a8230d797de3","Type":"ContainerDied","Data":"42a3731146bbe6a063650c66fff1b94a60f4977a0a7739114827e3551fd192e5"} Oct 29 23:45:30.102565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.102550 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42a3731146bbe6a063650c66fff1b94a60f4977a0a7739114827e3551fd192e5" Oct 29 23:45:30.110046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.109656 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:45:30.244336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.242722 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10608824-0cc9-41b0-b4a9-a8230d797de3-kube-api-access-pcxwc\" (UniqueName: \"kubernetes.io/projected/10608824-0cc9-41b0-b4a9-a8230d797de3-kube-api-access-pcxwc\") pod \"10608824-0cc9-41b0-b4a9-a8230d797de3\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " Oct 29 23:45:30.244336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.242894 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-maas-api-tls\") pod \"10608824-0cc9-41b0-b4a9-a8230d797de3\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " Oct 29 23:45:30.244336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.242984 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10608824-0cc9-41b0-b4a9-a8230d797de3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10608824-0cc9-41b0-b4a9-a8230d797de3-openshift-service-ca\") pod \"10608824-0cc9-41b0-b4a9-a8230d797de3\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " Oct 29 23:45:30.244336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.243078 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-metrics-certs\") pod \"10608824-0cc9-41b0-b4a9-a8230d797de3\" (UID: \"10608824-0cc9-41b0-b4a9-a8230d797de3\") " Oct 29 23:45:30.245527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.245502 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10608824-0cc9-41b0-b4a9-a8230d797de3-openshift-service-ca" pod "10608824-0cc9-41b0-b4a9-a8230d797de3" (UID: "10608824-0cc9-41b0-b4a9-a8230d797de3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:30.248662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.248561 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-metrics-certs" pod "10608824-0cc9-41b0-b4a9-a8230d797de3" (UID: "10608824-0cc9-41b0-b4a9-a8230d797de3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:30.250934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.250878 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-maas-api-tls" pod "10608824-0cc9-41b0-b4a9-a8230d797de3" (UID: "10608824-0cc9-41b0-b4a9-a8230d797de3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:30.251749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.251723 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10608824-0cc9-41b0-b4a9-a8230d797de3-kube-api-access-pcxwc" pod "10608824-0cc9-41b0-b4a9-a8230d797de3" (UID: "10608824-0cc9-41b0-b4a9-a8230d797de3"). InnerVolumeSpecName "kube-api-access-pcxwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:30.354938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.354837 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:30.354938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.354866 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10608824-0cc9-41b0-b4a9-a8230d797de3-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:30.354938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.354881 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10608824-0cc9-41b0-b4a9-a8230d797de3-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:30.354938 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.354896 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcxwc\" (UniqueName: \"kubernetes.io/projected/10608824-0cc9-41b0-b4a9-a8230d797de3-kube-api-access-pcxwc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:30.388522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.388493 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:30.442646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.442616 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb"] Oct 29 23:45:30.510815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.510727 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e10975-2xbzb"] Oct 29 23:45:30.569157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.567063 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-openshift-ca\") pod \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " Oct 29 23:45:30.569157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.567323 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5939aa-428e-49c3-bdd7-4982bb610cf1-kube-api-access-d5md5\" (UniqueName: \"kubernetes.io/projected/1b5939aa-428e-49c3-bdd7-4982bb610cf1-kube-api-access-d5md5\") pod \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " Oct 29 23:45:30.569157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.567424 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-xks-ca\") pod \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\" (UID: \"1b5939aa-428e-49c3-bdd7-4982bb610cf1\") " Oct 29 23:45:30.569157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.568902 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-openshift-ca" pod "1b5939aa-428e-49c3-bdd7-4982bb610cf1" (UID: "1b5939aa-428e-49c3-bdd7-4982bb610cf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:30.586631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.586599 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-xks-ca" pod "1b5939aa-428e-49c3-bdd7-4982bb610cf1" (UID: "1b5939aa-428e-49c3-bdd7-4982bb610cf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:30.587911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.587844 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5939aa-428e-49c3-bdd7-4982bb610cf1-kube-api-access-d5md5" pod "1b5939aa-428e-49c3-bdd7-4982bb610cf1" (UID: "1b5939aa-428e-49c3-bdd7-4982bb610cf1"). InnerVolumeSpecName "kube-api-access-d5md5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:30.796655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.719979 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5md5\" (UniqueName: \"kubernetes.io/projected/1b5939aa-428e-49c3-bdd7-4982bb610cf1-kube-api-access-d5md5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:30.796655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.767669 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:30.796655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:30.767702 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5939aa-428e-49c3-bdd7-4982bb610cf1-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.033629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.015110 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94abd8a1-519d-4c4a-8217-4b153bcc17a5" path="/var/lib/kubelet/pods/94abd8a1-519d-4c4a-8217-4b153bcc17a5/volumes" Oct 29 23:45:31.096856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.096822 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:45:31.110109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.110080 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" event={"ID":"ff57e2f5-af27-42fd-a5ed-0bd386e72344","Type":"ContainerStarted","Data":"441bd3e3594a0ce808e6b7acfdf1f3ea99226f2e0be730cfc78aef21ba285491"} Oct 29 23:45:31.111656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.111633 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" Oct 29 23:45:31.111791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.111678 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s" event={"ID":"1b5939aa-428e-49c3-bdd7-4982bb610cf1","Type":"ContainerDied","Data":"4ff2d27bb52857b90b8154a4d055f446c99c9601ee56210240ebce2505925871"} Oct 29 23:45:31.111791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.111721 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff2d27bb52857b90b8154a4d055f446c99c9601ee56210240ebce2505925871" Oct 29 23:45:31.113141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.113117 5736 generic.go:356] "Generic (PLEG): container finished" podID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerID="094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15" exitCode=0 Oct 29 23:45:31.113257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.113163 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" Oct 29 23:45:31.113257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.113212 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" event={"ID":"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1","Type":"ContainerDied","Data":"094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15"} Oct 29 23:45:31.113361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.113257 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz" event={"ID":"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1","Type":"ContainerDied","Data":"a9dcee5aa70d6579f32dea4befba6f92f7d802957edf66b99844e189f8868786"} Oct 29 23:45:31.113361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.113294 5736 scope.go:122] "RemoveContainer" containerID="094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15" Oct 29 23:45:31.113361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.113353 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" Oct 29 23:45:31.125283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.125266 5736 scope.go:122] "RemoveContainer" containerID="094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15" Oct 29 23:45:31.125559 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:31.125536 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15\": container with ID starting with 094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15 not found: ID does not exist" containerID="094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15" Oct 29 23:45:31.125662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.125561 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15"} err="failed to get container status \"094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15\": rpc error: code = NotFound desc = could not find container \"094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15\": container with ID starting with 094426c20d1c5c2bfe5b99789c8ce3b750d6f917b40b82b577f8e2f1fb551d15 not found: ID does not exist" Oct 29 23:45:31.194881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.194847 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-certs\") pod \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " Oct 29 23:45:31.195211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.195175 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-certs" pod "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" (UID: "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.195362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.195223 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-envoy\") pod \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " Oct 29 23:45:31.195362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.195290 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-credential-socket\") pod \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " Oct 29 23:45:31.195500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.195413 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-kube-api-access-xxwfb\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-kube-api-access-xxwfb\") pod \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " Oct 29 23:45:31.195500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.195456 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-podinfo\") pod \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " Oct 29 23:45:31.195730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.195564 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-token\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-token\") pod \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " Oct 29 23:45:31.195730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.195686 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-data\") pod \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " Oct 29 23:45:31.195868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.195761 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-credential-socket" pod "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" (UID: "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.195981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.195959 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-data" pod "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" (UID: "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.196094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.196048 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istiod-ca-cert\") pod \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " Oct 29 23:45:31.196466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.196419 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istiod-ca-cert" pod "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" (UID: "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:31.196689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.196667 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-socket\") pod \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\" (UID: \"2eed4fb6-f1b5-4402-8592-b4c98fcd31d1\") " Oct 29 23:45:31.196935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.196899 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-socket" pod "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" (UID: "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.198561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.198532 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-podinfo" pod "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" (UID: "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:45:31.198717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.198566 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-kube-api-access-xxwfb" pod "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" (UID: "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1"). InnerVolumeSpecName "kube-api-access-xxwfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:31.198793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.198726 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-token" pod "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" (UID: "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:31.198975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.198951 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxwfb\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-kube-api-access-xxwfb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.199046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.198980 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.199046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.198996 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.199046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.199010 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.199046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.199024 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.199046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.199038 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.199323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.199052 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.199323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.199066 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.199730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.199703 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-envoy" pod "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" (UID: "2eed4fb6-f1b5-4402-8592-b4c98fcd31d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:31.202111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.202060 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" podStartSLOduration=3.20204403 podStartE2EDuration="3.20204403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:31.200405265 +0000 UTC m=+3728.316324872" watchObservedRunningTime="2026-10-29 23:45:31.20204403 +0000 UTC m=+3728.317963636" Oct 29 23:45:31.242933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.242896 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr"] Oct 29 23:45:31.247439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.247406 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr"] Oct 29 23:45:31.300892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.300864 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:31.464671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.464643 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz"] Oct 29 23:45:31.468995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.468967 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b73a75-openshift-default-54957fbc85-hdzdz"] Oct 29 23:45:31.917914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.917873 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" podUID="d640b153-10af-4412-8334-97e3f60cb32f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:45:31.917914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:31.917898 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" podUID="88078911-4efa-4a76-82d1-70bce5369a7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:45:32.088933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:32.088896 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:32.090171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:32.090136 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" start-of-body= Oct 29 23:45:32.090345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:32.090190 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" Oct 29 23:45:32.942608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:32.940664 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e10975-6f9dd8898c-rtxtr" podUID="10608824-0cc9-41b0-b4a9-a8230d797de3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.12:8443/health\": context deadline exceeded" Oct 29 23:45:33.008281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:33.008244 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10608824-0cc9-41b0-b4a9-a8230d797de3" path="/var/lib/kubelet/pods/10608824-0cc9-41b0-b4a9-a8230d797de3/volumes" Oct 29 23:45:33.008856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:33.008840 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" path="/var/lib/kubelet/pods/2eed4fb6-f1b5-4402-8592-b4c98fcd31d1/volumes" Oct 29 23:45:33.093639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:33.091997 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" start-of-body= Oct 29 23:45:33.093639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:33.092078 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" Oct 29 23:45:34.086823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:34.086779 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" start-of-body= Oct 29 23:45:34.087246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:34.086840 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" Oct 29 23:45:35.086978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:35.086938 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" start-of-body= Oct 29 23:45:35.087379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:35.086994 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" Oct 29 23:45:36.086352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:36.086311 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" start-of-body= Oct 29 23:45:36.086535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:36.086370 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" Oct 29 23:45:36.660327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:36.659294 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx"] Oct 29 23:45:37.087458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:37.087410 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" start-of-body= Oct 29 23:45:37.087712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:37.087497 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.157:15021/healthz/ready\": dial tcp 10.134.1.157:15021: connect: connection refused" Oct 29 23:45:38.198510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.194164 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:38.198510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.196039 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:38.199125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.198530 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:45:38.766295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.766251 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm"] Oct 29 23:45:38.770609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.770569 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c3fc-h6qhm"] Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.891816 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch"] Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.893018 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.893045 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.893098 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1b5939aa-428e-49c3-bdd7-4982bb610cf1" containerName="revoke-keys" Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.893106 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5939aa-428e-49c3-bdd7-4982bb610cf1" containerName="revoke-keys" Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.893133 5736 cpu_manager.go:414] "containerMap: removing container" podUID="10608824-0cc9-41b0-b4a9-a8230d797de3" containerName="maas-api" Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.893142 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10608824-0cc9-41b0-b4a9-a8230d797de3" containerName="maas-api" Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.893351 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eed4fb6-f1b5-4402-8592-b4c98fcd31d1" containerName="istio-proxy" Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.893378 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5939aa-428e-49c3-bdd7-4982bb610cf1" containerName="revoke-keys" Oct 29 23:45:38.893600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.893391 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="10608824-0cc9-41b0-b4a9-a8230d797de3" containerName="maas-api" Oct 29 23:45:38.902553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.902513 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:38.915876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.915845 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch"] Oct 29 23:45:38.917764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.917736 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:45:38.917898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.917841 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:45:38.990270 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.990233 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:38.990520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.990493 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc8vz\" (UniqueName: \"kubernetes.io/projected/c26dfcc8-e552-4811-b74e-74bd5631dd7a-kube-api-access-gc8vz\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:38.990608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:38.990590 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:39.006864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:39.006829 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef13968-b3ee-4edc-b04b-a2104f8e4c58" path="/var/lib/kubelet/pods/6ef13968-b3ee-4edc-b04b-a2104f8e4c58/volumes" Oct 29 23:45:39.092918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:39.092308 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8vz\" (UniqueName: \"kubernetes.io/projected/c26dfcc8-e552-4811-b74e-74bd5631dd7a-kube-api-access-gc8vz\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:39.092918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:39.092523 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:39.092918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:39.092830 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:39.093479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:39.093459 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:39.094959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:39.094939 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:39.115802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:39.115777 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8vz\" (UniqueName: \"kubernetes.io/projected/c26dfcc8-e552-4811-b74e-74bd5631dd7a-kube-api-access-gc8vz\") pod \"maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:39.222817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:39.222787 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:39.539739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:39.539708 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch"] Oct 29 23:45:39.542322 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:39.542292 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26dfcc8_e552_4811_b74e_74bd5631dd7a.slice/crio-90dbe376c7016da30c858a2007ee979658a8153864f8af40d6bf5c2613e55240 WatchSource:0}: Error finding container 90dbe376c7016da30c858a2007ee979658a8153864f8af40d6bf5c2613e55240: Status 404 returned error can't find the container with id 90dbe376c7016da30c858a2007ee979658a8153864f8af40d6bf5c2613e55240 Oct 29 23:45:40.262949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:40.262912 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" event={"ID":"c26dfcc8-e552-4811-b74e-74bd5631dd7a","Type":"ContainerStarted","Data":"578d18be7c1cfa580cb2f58dd85874ef4b1defc25b44b4e9033c52536e472613"} Oct 29 23:45:40.263315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:40.262967 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" event={"ID":"c26dfcc8-e552-4811-b74e-74bd5631dd7a","Type":"ContainerStarted","Data":"90dbe376c7016da30c858a2007ee979658a8153864f8af40d6bf5c2613e55240"} Oct 29 23:45:40.304201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:40.304143 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" podStartSLOduration=2.3041258940000002 podStartE2EDuration="2.304125894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:40.302287708 +0000 UTC m=+3737.418207316" watchObservedRunningTime="2026-10-29 23:45:40.304125894 +0000 UTC m=+3737.420045500" Oct 29 23:45:41.274327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:41.274285 5736 generic.go:356] "Generic (PLEG): container finished" podID="c26dfcc8-e552-4811-b74e-74bd5631dd7a" containerID="578d18be7c1cfa580cb2f58dd85874ef4b1defc25b44b4e9033c52536e472613" exitCode=0 Oct 29 23:45:41.274826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:41.274371 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" event={"ID":"c26dfcc8-e552-4811-b74e-74bd5631dd7a","Type":"ContainerDied","Data":"578d18be7c1cfa580cb2f58dd85874ef4b1defc25b44b4e9033c52536e472613"} Oct 29 23:45:41.659946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:41.659861 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" podUID="3cb9bb9b-227a-41ad-8026-3ff61b4cc361" containerName="payload-processing" containerID="cri-o://607353011d7df5696d9a8b811779bde9788a4004359585f16799f86f86d1707f" gracePeriod=25 Oct 29 23:45:41.917291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:41.917192 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" podUID="88078911-4efa-4a76-82d1-70bce5369a7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:45:41.917291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:41.917236 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" podUID="d640b153-10af-4412-8334-97e3f60cb32f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:45:42.467609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.467570 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:42.567909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.567828 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26dfcc8-e552-4811-b74e-74bd5631dd7a-kube-api-access-gc8vz\" (UniqueName: \"kubernetes.io/projected/c26dfcc8-e552-4811-b74e-74bd5631dd7a-kube-api-access-gc8vz\") pod \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " Oct 29 23:45:42.568085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.568023 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-xks-ca\") pod \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " Oct 29 23:45:42.568287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.568271 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-openshift-ca\") pod \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\" (UID: \"c26dfcc8-e552-4811-b74e-74bd5631dd7a\") " Oct 29 23:45:42.568696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.568669 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-openshift-ca" pod "c26dfcc8-e552-4811-b74e-74bd5631dd7a" (UID: "c26dfcc8-e552-4811-b74e-74bd5631dd7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:42.568940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.568926 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:42.570018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.569990 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-xks-ca" pod "c26dfcc8-e552-4811-b74e-74bd5631dd7a" (UID: "c26dfcc8-e552-4811-b74e-74bd5631dd7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:42.570103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.570026 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26dfcc8-e552-4811-b74e-74bd5631dd7a-kube-api-access-gc8vz" pod "c26dfcc8-e552-4811-b74e-74bd5631dd7a" (UID: "c26dfcc8-e552-4811-b74e-74bd5631dd7a"). InnerVolumeSpecName "kube-api-access-gc8vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:42.670429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.670396 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc8vz\" (UniqueName: \"kubernetes.io/projected/c26dfcc8-e552-4811-b74e-74bd5631dd7a-kube-api-access-gc8vz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:42.670429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:42.670427 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c26dfcc8-e552-4811-b74e-74bd5631dd7a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:43.293650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:43.293537 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" Oct 29 23:45:43.293650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:43.293574 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch" event={"ID":"c26dfcc8-e552-4811-b74e-74bd5631dd7a","Type":"ContainerDied","Data":"90dbe376c7016da30c858a2007ee979658a8153864f8af40d6bf5c2613e55240"} Oct 29 23:45:43.293841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:43.293656 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90dbe376c7016da30c858a2007ee979658a8153864f8af40d6bf5c2613e55240" Oct 29 23:45:44.003137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.003098 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:45:44.043368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.043314 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:45:44.075164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.075130 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m"] Oct 29 23:45:44.075482 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.075406 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" podUID="dcda232c-83c2-4a12-9577-7daabecc5f22" containerName="maas-api" containerID="cri-o://c0b2a07b73bbb1bb9aabdfb80eb52e8be6c9a03fa8d229e29326b65d061cb5fa" gracePeriod=30 Oct 29 23:45:44.135603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.133533 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d640b153-10af-4412-8334-97e3f60cb32f-kube-api-access-9dgrb\" (UniqueName: \"kubernetes.io/projected/d640b153-10af-4412-8334-97e3f60cb32f-kube-api-access-9dgrb\") pod \"d640b153-10af-4412-8334-97e3f60cb32f\" (UID: \"d640b153-10af-4412-8334-97e3f60cb32f\") " Oct 29 23:45:44.135603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.133788 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d640b153-10af-4412-8334-97e3f60cb32f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d640b153-10af-4412-8334-97e3f60cb32f-plugins-config-volume\") pod \"d640b153-10af-4412-8334-97e3f60cb32f\" (UID: \"d640b153-10af-4412-8334-97e3f60cb32f\") " Oct 29 23:45:44.135603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.133947 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88078911-4efa-4a76-82d1-70bce5369a7c-kube-api-access-2r7m8\" (UniqueName: \"kubernetes.io/projected/88078911-4efa-4a76-82d1-70bce5369a7c-kube-api-access-2r7m8\") pod \"88078911-4efa-4a76-82d1-70bce5369a7c\" (UID: \"88078911-4efa-4a76-82d1-70bce5369a7c\") " Oct 29 23:45:44.135603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.134093 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88078911-4efa-4a76-82d1-70bce5369a7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88078911-4efa-4a76-82d1-70bce5369a7c-plugins-config-volume\") pod \"88078911-4efa-4a76-82d1-70bce5369a7c\" (UID: \"88078911-4efa-4a76-82d1-70bce5369a7c\") " Oct 29 23:45:44.135603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.134651 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88078911-4efa-4a76-82d1-70bce5369a7c-plugins-config-volume" pod "88078911-4efa-4a76-82d1-70bce5369a7c" (UID: "88078911-4efa-4a76-82d1-70bce5369a7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:44.135603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.135147 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88078911-4efa-4a76-82d1-70bce5369a7c-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.140006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.136066 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d640b153-10af-4412-8334-97e3f60cb32f-plugins-config-volume" pod "d640b153-10af-4412-8334-97e3f60cb32f" (UID: "d640b153-10af-4412-8334-97e3f60cb32f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:44.140006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.137033 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d640b153-10af-4412-8334-97e3f60cb32f-kube-api-access-9dgrb" pod "d640b153-10af-4412-8334-97e3f60cb32f" (UID: "d640b153-10af-4412-8334-97e3f60cb32f"). InnerVolumeSpecName "kube-api-access-9dgrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:44.140006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.138731 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88078911-4efa-4a76-82d1-70bce5369a7c-kube-api-access-2r7m8" pod "88078911-4efa-4a76-82d1-70bce5369a7c" (UID: "88078911-4efa-4a76-82d1-70bce5369a7c"). InnerVolumeSpecName "kube-api-access-2r7m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:44.236685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.236643 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d640b153-10af-4412-8334-97e3f60cb32f-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.236685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.236678 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r7m8\" (UniqueName: \"kubernetes.io/projected/88078911-4efa-4a76-82d1-70bce5369a7c-kube-api-access-2r7m8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.236685 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.236689 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dgrb\" (UniqueName: \"kubernetes.io/projected/d640b153-10af-4412-8334-97e3f60cb32f-kube-api-access-9dgrb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:44.607384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.605731 5736 generic.go:356] "Generic (PLEG): container finished" podID="d640b153-10af-4412-8334-97e3f60cb32f" containerID="afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c" exitCode=0 Oct 29 23:45:44.607384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.605991 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" event={"ID":"d640b153-10af-4412-8334-97e3f60cb32f","Type":"ContainerDied","Data":"afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c"} Oct 29 23:45:44.607384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.606048 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" event={"ID":"d640b153-10af-4412-8334-97e3f60cb32f","Type":"ContainerDied","Data":"3be985af721fd7ca00587047b398d92613ef61fd46b3ebd50b5071484b3ce76e"} Oct 29 23:45:44.607384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.606080 5736 scope.go:122] "RemoveContainer" containerID="afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c" Oct 29 23:45:44.607384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.606236 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw" Oct 29 23:45:44.615664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.615617 5736 generic.go:356] "Generic (PLEG): container finished" podID="dcda232c-83c2-4a12-9577-7daabecc5f22" containerID="c0b2a07b73bbb1bb9aabdfb80eb52e8be6c9a03fa8d229e29326b65d061cb5fa" exitCode=0 Oct 29 23:45:44.616017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.615990 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" event={"ID":"dcda232c-83c2-4a12-9577-7daabecc5f22","Type":"ContainerDied","Data":"c0b2a07b73bbb1bb9aabdfb80eb52e8be6c9a03fa8d229e29326b65d061cb5fa"} Oct 29 23:45:44.622835 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.622803 5736 generic.go:356] "Generic (PLEG): container finished" podID="88078911-4efa-4a76-82d1-70bce5369a7c" containerID="adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226" exitCode=0 Oct 29 23:45:44.623044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.623022 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" event={"ID":"88078911-4efa-4a76-82d1-70bce5369a7c","Type":"ContainerDied","Data":"adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226"} Oct 29 23:45:44.623681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.623655 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" event={"ID":"88078911-4efa-4a76-82d1-70bce5369a7c","Type":"ContainerDied","Data":"2c84e64965f84a82bfc07aac58589ed27a6cd2a046908f7e887d6b19a53b7c4c"} Oct 29 23:45:44.623896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.623880 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n" Oct 29 23:45:44.852602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.845705 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:45:44.880033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.879915 5736 scope.go:122] "RemoveContainer" containerID="afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c" Oct 29 23:45:44.880760 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:44.880589 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c\": container with ID starting with afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c not found: ID does not exist" containerID="afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c" Oct 29 23:45:44.880760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.880631 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c"} err="failed to get container status \"afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c\": rpc error: code = NotFound desc = could not find container \"afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c\": container with ID starting with afe9a89448f4cc886661dfd5c59d8addc7be155e442f220683fd069caf8f2a0c not found: ID does not exist" Oct 29 23:45:44.880760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.880661 5736 scope.go:122] "RemoveContainer" containerID="adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226" Oct 29 23:45:44.930811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.930772 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n"] Oct 29 23:45:44.975285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.943297 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-metrics-certs\") pod \"dcda232c-83c2-4a12-9577-7daabecc5f22\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " Oct 29 23:45:44.975285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.943403 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcda232c-83c2-4a12-9577-7daabecc5f22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcda232c-83c2-4a12-9577-7daabecc5f22-openshift-service-ca\") pod \"dcda232c-83c2-4a12-9577-7daabecc5f22\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " Oct 29 23:45:44.975285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.943514 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-maas-api-tls\") pod \"dcda232c-83c2-4a12-9577-7daabecc5f22\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " Oct 29 23:45:44.975285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.943612 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcda232c-83c2-4a12-9577-7daabecc5f22-kube-api-access-v7nfm\" (UniqueName: \"kubernetes.io/projected/dcda232c-83c2-4a12-9577-7daabecc5f22-kube-api-access-v7nfm\") pod \"dcda232c-83c2-4a12-9577-7daabecc5f22\" (UID: \"dcda232c-83c2-4a12-9577-7daabecc5f22\") " Oct 29 23:45:44.975285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.957077 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcda232c-83c2-4a12-9577-7daabecc5f22-openshift-service-ca" pod "dcda232c-83c2-4a12-9577-7daabecc5f22" (UID: "dcda232c-83c2-4a12-9577-7daabecc5f22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:44.978165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:44.978129 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcda232c-83c2-4a12-9577-7daabecc5f22-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:45.009752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.009713 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-maas-api-tls" pod "dcda232c-83c2-4a12-9577-7daabecc5f22" (UID: "dcda232c-83c2-4a12-9577-7daabecc5f22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:45.017626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.017595 5736 scope.go:122] "RemoveContainer" containerID="adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226" Oct 29 23:45:45.038638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.035771 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-metrics-certs" pod "dcda232c-83c2-4a12-9577-7daabecc5f22" (UID: "dcda232c-83c2-4a12-9577-7daabecc5f22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:45.044372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.043515 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcda232c-83c2-4a12-9577-7daabecc5f22-kube-api-access-v7nfm" pod "dcda232c-83c2-4a12-9577-7daabecc5f22" (UID: "dcda232c-83c2-4a12-9577-7daabecc5f22"). InnerVolumeSpecName "kube-api-access-v7nfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:45.051672 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:45.051634 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226\": container with ID starting with adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226 not found: ID does not exist" containerID="adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226" Oct 29 23:45:45.051869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.051837 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226"} err="failed to get container status \"adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226\": rpc error: code = NotFound desc = could not find container \"adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226\": container with ID starting with adc61396a800e5eba7975613fbd93356bd96233ebd9ecc25863bb68edf793226 not found: ID does not exist" Oct 29 23:45:45.055707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.055681 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51462d-79cb6d9c7b-lwf7n"] Oct 29 23:45:45.077497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.077471 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw"] Oct 29 23:45:45.079770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.079324 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:45.079770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.079351 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nfm\" (UniqueName: \"kubernetes.io/projected/dcda232c-83c2-4a12-9577-7daabecc5f22-kube-api-access-v7nfm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:45.079770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.079366 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcda232c-83c2-4a12-9577-7daabecc5f22-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:45.083819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.083790 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51462d-857f57d667-b8rtw"] Oct 29 23:45:45.107118 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:45.107087 5736 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88078911_4efa_4a76_82d1_70bce5369a7c.slice/crio-2c84e64965f84a82bfc07aac58589ed27a6cd2a046908f7e887d6b19a53b7c4c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88078911_4efa_4a76_82d1_70bce5369a7c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcda232c_83c2_4a12_9577_7daabecc5f22.slice/crio-b6976d5ec9967ab62a454dac40255b2b7904f4d02588d33ad2a0cb47155e47cd\": RecentStats: unable to find data in memory cache]" Oct 29 23:45:45.191664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.188049 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s"] Oct 29 23:45:45.194159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.194122 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c83b50-w9z6s"] Oct 29 23:45:45.632705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.632669 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" Oct 29 23:45:45.632912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.632676 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m" event={"ID":"dcda232c-83c2-4a12-9577-7daabecc5f22","Type":"ContainerDied","Data":"b6976d5ec9967ab62a454dac40255b2b7904f4d02588d33ad2a0cb47155e47cd"} Oct 29 23:45:45.632912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.632823 5736 scope.go:122] "RemoveContainer" containerID="c0b2a07b73bbb1bb9aabdfb80eb52e8be6c9a03fa8d229e29326b65d061cb5fa" Oct 29 23:45:45.688932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.688904 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m"] Oct 29 23:45:45.692772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:45.692747 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c83b50-5944dfd6f8-ln29m"] Oct 29 23:45:46.086076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:46.086035 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" podUID="3cb9bb9b-227a-41ad-8026-3ff61b4cc361" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:45:46.700400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:46.698245 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm"] Oct 29 23:45:46.700400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:46.698496 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" containerID="cri-o://6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899" gracePeriod=30 Oct 29 23:45:47.044603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.043953 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5939aa-428e-49c3-bdd7-4982bb610cf1" path="/var/lib/kubelet/pods/1b5939aa-428e-49c3-bdd7-4982bb610cf1/volumes" Oct 29 23:45:47.044805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.044733 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88078911-4efa-4a76-82d1-70bce5369a7c" path="/var/lib/kubelet/pods/88078911-4efa-4a76-82d1-70bce5369a7c/volumes" Oct 29 23:45:47.051909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.045285 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d640b153-10af-4412-8334-97e3f60cb32f" path="/var/lib/kubelet/pods/d640b153-10af-4412-8334-97e3f60cb32f/volumes" Oct 29 23:45:47.051909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.047478 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcda232c-83c2-4a12-9577-7daabecc5f22" path="/var/lib/kubelet/pods/dcda232c-83c2-4a12-9577-7daabecc5f22/volumes" Oct 29 23:45:47.412251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.412174 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh"] Oct 29 23:45:47.412926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.412910 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c26dfcc8-e552-4811-b74e-74bd5631dd7a" containerName="revoke-keys" Oct 29 23:45:47.412993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.412928 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26dfcc8-e552-4811-b74e-74bd5631dd7a" containerName="revoke-keys" Oct 29 23:45:47.412993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.412945 5736 cpu_manager.go:414] "containerMap: removing container" podUID="dcda232c-83c2-4a12-9577-7daabecc5f22" containerName="maas-api" Oct 29 23:45:47.412993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.412950 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcda232c-83c2-4a12-9577-7daabecc5f22" containerName="maas-api" Oct 29 23:45:47.413100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.413011 5736 cpu_manager.go:414] "containerMap: removing container" podUID="88078911-4efa-4a76-82d1-70bce5369a7c" containerName="payload-processing" Oct 29 23:45:47.413100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.413018 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88078911-4efa-4a76-82d1-70bce5369a7c" containerName="payload-processing" Oct 29 23:45:47.413100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.413032 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d640b153-10af-4412-8334-97e3f60cb32f" containerName="payload-pre-processing" Oct 29 23:45:47.413100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.413039 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d640b153-10af-4412-8334-97e3f60cb32f" containerName="payload-pre-processing" Oct 29 23:45:47.413232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.413151 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="88078911-4efa-4a76-82d1-70bce5369a7c" containerName="payload-processing" Oct 29 23:45:47.413232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.413163 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d640b153-10af-4412-8334-97e3f60cb32f" containerName="payload-pre-processing" Oct 29 23:45:47.413232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.413182 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26dfcc8-e552-4811-b74e-74bd5631dd7a" containerName="revoke-keys" Oct 29 23:45:47.413232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.413190 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcda232c-83c2-4a12-9577-7daabecc5f22" containerName="maas-api" Oct 29 23:45:47.418572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.418551 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.429975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.429951 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzcgx\" (UniqueName: \"kubernetes.io/projected/41d66738-1738-4538-9efb-31db79c154e8-kube-api-access-jzcgx\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.430070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.430030 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d66738-1738-4538-9efb-31db79c154e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.430227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.430198 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d66738-1738-4538-9efb-31db79c154e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.438651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.438628 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh"] Oct 29 23:45:47.484087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.484032 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:45:47.484408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.484383 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:45:47.531286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.531255 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzcgx\" (UniqueName: \"kubernetes.io/projected/41d66738-1738-4538-9efb-31db79c154e8-kube-api-access-jzcgx\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.531422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.531338 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d66738-1738-4538-9efb-31db79c154e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.531422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.531408 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d66738-1738-4538-9efb-31db79c154e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.532009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.531987 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d66738-1738-4538-9efb-31db79c154e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.533982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.533956 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d66738-1738-4538-9efb-31db79c154e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.580147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.574286 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzcgx\" (UniqueName: \"kubernetes.io/projected/41d66738-1738-4538-9efb-31db79c154e8-kube-api-access-jzcgx\") pod \"maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.733870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.733838 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:47.770054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.770018 5736 patch_prober.go:29] interesting pod/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:45:47.770192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:47.770084 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:45:48.044812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:48.044781 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh"] Oct 29 23:45:48.046279 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:48.046250 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d66738_1738_4538_9efb_31db79c154e8.slice/crio-43db47cb7251d76d6561ca017c26a7f393eb08a9b0a6d553690ff3579d594486 WatchSource:0}: Error finding container 43db47cb7251d76d6561ca017c26a7f393eb08a9b0a6d553690ff3579d594486: Status 404 returned error can't find the container with id 43db47cb7251d76d6561ca017c26a7f393eb08a9b0a6d553690ff3579d594486 Oct 29 23:45:48.670670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:48.670540 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" event={"ID":"41d66738-1738-4538-9efb-31db79c154e8","Type":"ContainerStarted","Data":"377f132a84020570770aa49474c9f1e7087602366ca43a49f3d5cd0e0dce0f8e"} Oct 29 23:45:48.670670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:48.670659 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" event={"ID":"41d66738-1738-4538-9efb-31db79c154e8","Type":"ContainerStarted","Data":"43db47cb7251d76d6561ca017c26a7f393eb08a9b0a6d553690ff3579d594486"} Oct 29 23:45:48.714843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:48.714786 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" podStartSLOduration=1.714773379 podStartE2EDuration="1.714773379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:48.710257347 +0000 UTC m=+3745.826176952" watchObservedRunningTime="2026-10-29 23:45:48.714773379 +0000 UTC m=+3745.830692984" Oct 29 23:45:49.590218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:49.590177 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth"] Oct 29 23:45:49.684138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:49.684092 5736 generic.go:356] "Generic (PLEG): container finished" podID="41d66738-1738-4538-9efb-31db79c154e8" containerID="377f132a84020570770aa49474c9f1e7087602366ca43a49f3d5cd0e0dce0f8e" exitCode=0 Oct 29 23:45:49.684646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:49.684161 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" event={"ID":"41d66738-1738-4538-9efb-31db79c154e8","Type":"ContainerDied","Data":"377f132a84020570770aa49474c9f1e7087602366ca43a49f3d5cd0e0dce0f8e"} Oct 29 23:45:50.871323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:50.871296 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:50.995430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:50.995397 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d66738-1738-4538-9efb-31db79c154e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d66738-1738-4538-9efb-31db79c154e8-maas-api-openshift-ca\") pod \"41d66738-1738-4538-9efb-31db79c154e8\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " Oct 29 23:45:50.995649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:50.995444 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d66738-1738-4538-9efb-31db79c154e8-kube-api-access-jzcgx\" (UniqueName: \"kubernetes.io/projected/41d66738-1738-4538-9efb-31db79c154e8-kube-api-access-jzcgx\") pod \"41d66738-1738-4538-9efb-31db79c154e8\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " Oct 29 23:45:50.995726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:50.995704 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d66738-1738-4538-9efb-31db79c154e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d66738-1738-4538-9efb-31db79c154e8-maas-api-xks-ca\") pod \"41d66738-1738-4538-9efb-31db79c154e8\" (UID: \"41d66738-1738-4538-9efb-31db79c154e8\") " Oct 29 23:45:50.995796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:50.995754 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d66738-1738-4538-9efb-31db79c154e8-maas-api-openshift-ca" pod "41d66738-1738-4538-9efb-31db79c154e8" (UID: "41d66738-1738-4538-9efb-31db79c154e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:50.996276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:50.996254 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d66738-1738-4538-9efb-31db79c154e8-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:50.997717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:50.997692 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d66738-1738-4538-9efb-31db79c154e8-maas-api-xks-ca" pod "41d66738-1738-4538-9efb-31db79c154e8" (UID: "41d66738-1738-4538-9efb-31db79c154e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:50.997821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:50.997723 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d66738-1738-4538-9efb-31db79c154e8-kube-api-access-jzcgx" pod "41d66738-1738-4538-9efb-31db79c154e8" (UID: "41d66738-1738-4538-9efb-31db79c154e8"). InnerVolumeSpecName "kube-api-access-jzcgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:51.098243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:51.098208 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d66738-1738-4538-9efb-31db79c154e8-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:51.098406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:51.098311 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzcgx\" (UniqueName: \"kubernetes.io/projected/41d66738-1738-4538-9efb-31db79c154e8-kube-api-access-jzcgx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:51.702699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:51.702664 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" Oct 29 23:45:51.702857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:51.702698 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh" event={"ID":"41d66738-1738-4538-9efb-31db79c154e8","Type":"ContainerDied","Data":"43db47cb7251d76d6561ca017c26a7f393eb08a9b0a6d553690ff3579d594486"} Oct 29 23:45:51.702857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:51.702748 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43db47cb7251d76d6561ca017c26a7f393eb08a9b0a6d553690ff3579d594486" Oct 29 23:45:52.063110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.063079 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:45:52.123879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123183 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-socket\") pod \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " Oct 29 23:45:52.123879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123253 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-token\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-token\") pod \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " Oct 29 23:45:52.123879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123301 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-podinfo\") pod \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " Oct 29 23:45:52.123879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123364 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-credential-socket\") pod \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " Oct 29 23:45:52.123879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123499 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-socket" pod "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" (UID: "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:52.123879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123653 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-data\") pod \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " Oct 29 23:45:52.123879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123704 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-kube-api-access-sjd4j\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-kube-api-access-sjd4j\") pod \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " Oct 29 23:45:52.123879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123773 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-envoy\") pod \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " Oct 29 23:45:52.123879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123871 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istiod-ca-cert\") pod \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " Oct 29 23:45:52.124782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123926 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-certs\") pod \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\" (UID: \"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2\") " Oct 29 23:45:52.124782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.123937 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-data" pod "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" (UID: "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:52.124782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.124262 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-credential-socket" pod "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" (UID: "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:52.124782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.124483 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-certs" pod "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" (UID: "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:52.127037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.124832 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:52.127037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.124854 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:52.127037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.124868 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:52.127037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.124882 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:52.127309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.127289 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-kube-api-access-sjd4j" pod "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" (UID: "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2"). InnerVolumeSpecName "kube-api-access-sjd4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:52.127867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.127570 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istiod-ca-cert" pod "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" (UID: "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:52.128865 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.128823 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-envoy" pod "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" (UID: "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:45:52.130969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.130944 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-podinfo" pod "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" (UID: "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:45:52.132211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.132185 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-token" pod "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" (UID: "f0946c19-9c6c-4fdc-bac2-cb943f88d8c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:52.226541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.226511 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:52.226541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.226539 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:52.226541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.226549 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:52.226791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.226557 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjd4j\" (UniqueName: \"kubernetes.io/projected/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-kube-api-access-sjd4j\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:52.226791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.226567 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:52.771619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.771546 5736 generic.go:356] "Generic (PLEG): container finished" podID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerID="6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899" exitCode=0 Oct 29 23:45:52.772129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.772105 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" Oct 29 23:45:52.773934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.773644 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" event={"ID":"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2","Type":"ContainerDied","Data":"6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899"} Oct 29 23:45:52.773934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.773718 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm" event={"ID":"f0946c19-9c6c-4fdc-bac2-cb943f88d8c2","Type":"ContainerDied","Data":"bc712b4d2a27cb66dcd9c33189c116022970b7f28e8c8926274be4d3ba47c9ef"} Oct 29 23:45:52.773934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.773750 5736 scope.go:122] "RemoveContainer" containerID="6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899" Oct 29 23:45:52.801860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.801832 5736 scope.go:122] "RemoveContainer" containerID="3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5" Oct 29 23:45:52.814676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.814472 5736 scope.go:122] "RemoveContainer" containerID="6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899" Oct 29 23:45:52.815074 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:52.815037 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899\": container with ID starting with 6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899 not found: ID does not exist" containerID="6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899" Oct 29 23:45:52.815182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.815085 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899"} err="failed to get container status \"6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899\": rpc error: code = NotFound desc = could not find container \"6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899\": container with ID starting with 6c0fdf3ecb3595797653c5d766bb5f7d1a05cb0e53dd823116afa42c2f2b4899 not found: ID does not exist" Oct 29 23:45:52.815182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.815164 5736 scope.go:122] "RemoveContainer" containerID="3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5" Oct 29 23:45:52.815541 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:52.815516 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5\": container with ID starting with 3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5 not found: ID does not exist" containerID="3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5" Oct 29 23:45:52.815639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.815548 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5"} err="failed to get container status \"3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5\": rpc error: code = NotFound desc = could not find container \"3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5\": container with ID starting with 3aadb8377d698b2f797838e81006bbbb31d9c48ccd358fbab7b8ce2c5eb37de5 not found: ID does not exist" Oct 29 23:45:52.852619 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.852571 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm"] Oct 29 23:45:52.862959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:52.862933 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-55c3fc-openshift-default-7594bf6dff-z2dvm"] Oct 29 23:45:53.008067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.008031 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" path="/var/lib/kubelet/pods/f0946c19-9c6c-4fdc-bac2-cb943f88d8c2/volumes" Oct 29 23:45:53.223836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.223802 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr"] Oct 29 23:45:53.224543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.224526 5736 cpu_manager.go:414] "containerMap: removing container" podUID="41d66738-1738-4538-9efb-31db79c154e8" containerName="revoke-keys" Oct 29 23:45:53.224625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.224544 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d66738-1738-4538-9efb-31db79c154e8" containerName="revoke-keys" Oct 29 23:45:53.224625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.224557 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" Oct 29 23:45:53.224625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.224562 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" Oct 29 23:45:53.224625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.224601 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" Oct 29 23:45:53.224625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.224610 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" Oct 29 23:45:53.224799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.224773 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" Oct 29 23:45:53.224799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.224791 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d66738-1738-4538-9efb-31db79c154e8" containerName="revoke-keys" Oct 29 23:45:53.230026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.229998 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.246265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.245966 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:45:53.246265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.246071 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:45:53.249867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.249843 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr"] Oct 29 23:45:53.282998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.282960 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.283200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.283021 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7n2j\" (UniqueName: \"kubernetes.io/projected/f0996c41-d830-47b5-8e47-d5e41ea27717-kube-api-access-l7n2j\") pod \"maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.283282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.283209 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.385125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.385085 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.385311 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.385284 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.385354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.385327 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7n2j\" (UniqueName: \"kubernetes.io/projected/f0996c41-d830-47b5-8e47-d5e41ea27717-kube-api-access-l7n2j\") pod \"maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.385995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.385964 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.387490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.387469 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.414423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.414393 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7n2j\" (UniqueName: \"kubernetes.io/projected/f0996c41-d830-47b5-8e47-d5e41ea27717-kube-api-access-l7n2j\") pod \"maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.545414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.545341 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:53.854903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:53.854869 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr"] Oct 29 23:45:53.857750 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:53.857709 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0996c41_d830_47b5_8e47_d5e41ea27717.slice/crio-f425b718af6bc7068efd56cb92127c1bc08754c0fd534ab2c97450ec7ab87122 WatchSource:0}: Error finding container f425b718af6bc7068efd56cb92127c1bc08754c0fd534ab2c97450ec7ab87122: Status 404 returned error can't find the container with id f425b718af6bc7068efd56cb92127c1bc08754c0fd534ab2c97450ec7ab87122 Oct 29 23:45:54.591023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:54.590907 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" podUID="1fcf1dc2-3f96-49df-9d31-a838a398ef06" containerName="payload-processing" containerID="cri-o://7697da753d9347e1186a24ba39acd495be8b275331ca9932aaf0d4bc3caffb91" gracePeriod=25 Oct 29 23:45:54.804081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:54.804042 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" event={"ID":"f0996c41-d830-47b5-8e47-d5e41ea27717","Type":"ContainerStarted","Data":"d3c01922355044520cf1e8cbd15cf4d84e2bc486618af8dff014e6c8e3b9a4e6"} Oct 29 23:45:54.804247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:54.804101 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" event={"ID":"f0996c41-d830-47b5-8e47-d5e41ea27717","Type":"ContainerStarted","Data":"f425b718af6bc7068efd56cb92127c1bc08754c0fd534ab2c97450ec7ab87122"} Oct 29 23:45:54.807196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:54.807147 5736 generic.go:356] "Generic (PLEG): container finished" podID="1fcf1dc2-3f96-49df-9d31-a838a398ef06" containerID="7697da753d9347e1186a24ba39acd495be8b275331ca9932aaf0d4bc3caffb91" exitCode=0 Oct 29 23:45:54.807350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:54.807212 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" event={"ID":"1fcf1dc2-3f96-49df-9d31-a838a398ef06","Type":"ContainerDied","Data":"7697da753d9347e1186a24ba39acd495be8b275331ca9932aaf0d4bc3caffb91"} Oct 29 23:45:54.846649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:54.845414 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" podStartSLOduration=1.845397067 podStartE2EDuration="1.845397067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:54.844271488 +0000 UTC m=+3751.960191095" watchObservedRunningTime="2026-10-29 23:45:54.845397067 +0000 UTC m=+3751.961316674" Oct 29 23:45:54.980369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:54.980335 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:45:55.122194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.122058 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fcf1dc2-3f96-49df-9d31-a838a398ef06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf1dc2-3f96-49df-9d31-a838a398ef06-plugins-config-volume\") pod \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\" (UID: \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\") " Oct 29 23:45:55.122194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.122141 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fcf1dc2-3f96-49df-9d31-a838a398ef06-kube-api-access-hzdrg\" (UniqueName: \"kubernetes.io/projected/1fcf1dc2-3f96-49df-9d31-a838a398ef06-kube-api-access-hzdrg\") pod \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\" (UID: \"1fcf1dc2-3f96-49df-9d31-a838a398ef06\") " Oct 29 23:45:55.122633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.122561 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcf1dc2-3f96-49df-9d31-a838a398ef06-plugins-config-volume" pod "1fcf1dc2-3f96-49df-9d31-a838a398ef06" (UID: "1fcf1dc2-3f96-49df-9d31-a838a398ef06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:55.123079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.123054 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf1dc2-3f96-49df-9d31-a838a398ef06-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:55.127759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.127729 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcf1dc2-3f96-49df-9d31-a838a398ef06-kube-api-access-hzdrg" pod "1fcf1dc2-3f96-49df-9d31-a838a398ef06" (UID: "1fcf1dc2-3f96-49df-9d31-a838a398ef06"). InnerVolumeSpecName "kube-api-access-hzdrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:55.224511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.224476 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzdrg\" (UniqueName: \"kubernetes.io/projected/1fcf1dc2-3f96-49df-9d31-a838a398ef06-kube-api-access-hzdrg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:55.595574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.595536 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49"] Oct 29 23:45:55.596811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.596771 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1fcf1dc2-3f96-49df-9d31-a838a398ef06" containerName="payload-processing" Oct 29 23:45:55.596811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.596799 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fcf1dc2-3f96-49df-9d31-a838a398ef06" containerName="payload-processing" Oct 29 23:45:55.599445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.597001 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fcf1dc2-3f96-49df-9d31-a838a398ef06" containerName="payload-processing" Oct 29 23:45:55.599445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.597024 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0946c19-9c6c-4fdc-bac2-cb943f88d8c2" containerName="istio-proxy" Oct 29 23:45:55.602247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.602225 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.622811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.622767 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49"] Oct 29 23:45:55.628889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.628865 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-37498a\"/\"openshift-service-ca.crt\"" Oct 29 23:45:55.629354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.629335 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-37498a\"/\"default-dockercfg-dmvv4\"" Oct 29 23:45:55.632098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.632073 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-37498a\"/\"kube-root-ca.crt\"" Oct 29 23:45:55.632307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.632290 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-37498a\"/\"test-model-37498a-kserve-self-signed-certs\"" Oct 29 23:45:55.732514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.732473 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-dshm\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.732694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.732552 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b131f7e8-db27-4412-9886-6556853efcba-tls-certs\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.732762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.732683 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-tmp-dir\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.732888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.732859 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9txj\" (UniqueName: \"kubernetes.io/projected/b131f7e8-db27-4412-9886-6556853efcba-kube-api-access-b9txj\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.732933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.732905 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-home\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.733002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.732978 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-model-cache\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.820090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.820053 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" Oct 29 23:45:55.820298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.820062 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth" event={"ID":"1fcf1dc2-3f96-49df-9d31-a838a398ef06","Type":"ContainerDied","Data":"f473b5ce486a3464af5b2bf8003cf2f2daae58936f257090abfa23c57dc3d8be"} Oct 29 23:45:55.820298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.820251 5736 scope.go:122] "RemoveContainer" containerID="7697da753d9347e1186a24ba39acd495be8b275331ca9932aaf0d4bc3caffb91" Oct 29 23:45:55.822355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.822314 5736 generic.go:356] "Generic (PLEG): container finished" podID="f0996c41-d830-47b5-8e47-d5e41ea27717" containerID="d3c01922355044520cf1e8cbd15cf4d84e2bc486618af8dff014e6c8e3b9a4e6" exitCode=0 Oct 29 23:45:55.822485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.822457 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" event={"ID":"f0996c41-d830-47b5-8e47-d5e41ea27717","Type":"ContainerDied","Data":"d3c01922355044520cf1e8cbd15cf4d84e2bc486618af8dff014e6c8e3b9a4e6"} Oct 29 23:45:55.839819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.836115 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9txj\" (UniqueName: \"kubernetes.io/projected/b131f7e8-db27-4412-9886-6556853efcba-kube-api-access-b9txj\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.839819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.836174 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-home\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.839819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.836215 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-model-cache\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.839819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.836493 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-dshm\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.839819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.836551 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b131f7e8-db27-4412-9886-6556853efcba-tls-certs\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.839819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.836620 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-tmp-dir\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.839819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.837079 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-tmp-dir\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.839819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.837779 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-home\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.866604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.864051 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-model-cache\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.896747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.893776 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9txj\" (UniqueName: \"kubernetes.io/projected/b131f7e8-db27-4412-9886-6556853efcba-kube-api-access-b9txj\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.962602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.961558 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth"] Oct 29 23:45:55.970607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.966264 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b131f7e8-db27-4412-9886-6556853efcba-tls-certs\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:55.970607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:55.966499 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-dshm\") pod \"test-model-37498a-kserve-c58d954f5-hzm49\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:56.070675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:56.055675 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7d4110-f6887874d-xjrth"] Oct 29 23:45:56.105674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:56.105158 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" podUID="3cb9bb9b-227a-41ad-8026-3ff61b4cc361" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:45:56.219879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:56.219834 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:56.542240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:56.542211 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49"] Oct 29 23:45:56.544964 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:45:56.544929 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb131f7e8_db27_4412_9886_6556853efcba.slice/crio-3f04bc10752c829c8c5f9266713bd91f5a2e56de93ee8fcad99c0f925bfa36b8 WatchSource:0}: Error finding container 3f04bc10752c829c8c5f9266713bd91f5a2e56de93ee8fcad99c0f925bfa36b8: Status 404 returned error can't find the container with id 3f04bc10752c829c8c5f9266713bd91f5a2e56de93ee8fcad99c0f925bfa36b8 Oct 29 23:45:56.826619 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:45:56.826520 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:45:56.834504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:56.834468 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" event={"ID":"b131f7e8-db27-4412-9886-6556853efcba","Type":"ContainerStarted","Data":"8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8"} Oct 29 23:45:56.834725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:56.834517 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" event={"ID":"b131f7e8-db27-4412-9886-6556853efcba","Type":"ContainerStarted","Data":"3f04bc10752c829c8c5f9266713bd91f5a2e56de93ee8fcad99c0f925bfa36b8"} Oct 29 23:45:56.878150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:56.878090 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" podStartSLOduration=1.8780736070000001 podStartE2EDuration="1.878073607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:56.875387766 +0000 UTC m=+3753.991307371" watchObservedRunningTime="2026-10-29 23:45:56.878073607 +0000 UTC m=+3753.993993212" Oct 29 23:45:57.059715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.059443 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fcf1dc2-3f96-49df-9d31-a838a398ef06" path="/var/lib/kubelet/pods/1fcf1dc2-3f96-49df-9d31-a838a398ef06/volumes" Oct 29 23:45:57.061491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.060947 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:57.168907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.168813 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-openshift-ca\") pod \"f0996c41-d830-47b5-8e47-d5e41ea27717\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " Oct 29 23:45:57.168907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.168871 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0996c41-d830-47b5-8e47-d5e41ea27717-kube-api-access-l7n2j\" (UniqueName: \"kubernetes.io/projected/f0996c41-d830-47b5-8e47-d5e41ea27717-kube-api-access-l7n2j\") pod \"f0996c41-d830-47b5-8e47-d5e41ea27717\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " Oct 29 23:45:57.169201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.169026 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-xks-ca\") pod \"f0996c41-d830-47b5-8e47-d5e41ea27717\" (UID: \"f0996c41-d830-47b5-8e47-d5e41ea27717\") " Oct 29 23:45:57.169276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.169247 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-openshift-ca" pod "f0996c41-d830-47b5-8e47-d5e41ea27717" (UID: "f0996c41-d830-47b5-8e47-d5e41ea27717"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:45:57.169863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.169840 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.171232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.171205 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0996c41-d830-47b5-8e47-d5e41ea27717-kube-api-access-l7n2j" pod "f0996c41-d830-47b5-8e47-d5e41ea27717" (UID: "f0996c41-d830-47b5-8e47-d5e41ea27717"). InnerVolumeSpecName "kube-api-access-l7n2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:45:57.171337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.171308 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-xks-ca" pod "f0996c41-d830-47b5-8e47-d5e41ea27717" (UID: "f0996c41-d830-47b5-8e47-d5e41ea27717"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:45:57.220977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.220944 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:45:57.271779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.271736 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7n2j\" (UniqueName: \"kubernetes.io/projected/f0996c41-d830-47b5-8e47-d5e41ea27717-kube-api-access-l7n2j\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.271779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.271778 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0996c41-d830-47b5-8e47-d5e41ea27717-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:45:57.848815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.848783 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" Oct 29 23:45:57.849229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.848806 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr" event={"ID":"f0996c41-d830-47b5-8e47-d5e41ea27717","Type":"ContainerDied","Data":"f425b718af6bc7068efd56cb92127c1bc08754c0fd534ab2c97450ec7ab87122"} Oct 29 23:45:57.849229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:57.848860 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f425b718af6bc7068efd56cb92127c1bc08754c0fd534ab2c97450ec7ab87122" Oct 29 23:45:58.012819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:45:58.012786 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56"] Oct 29 23:46:01.918550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:01.918319 5736 generic.go:356] "Generic (PLEG): container finished" podID="3cb9bb9b-227a-41ad-8026-3ff61b4cc361" containerID="607353011d7df5696d9a8b811779bde9788a4004359585f16799f86f86d1707f" exitCode=0 Oct 29 23:46:01.918550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:01.918358 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" event={"ID":"3cb9bb9b-227a-41ad-8026-3ff61b4cc361","Type":"ContainerDied","Data":"607353011d7df5696d9a8b811779bde9788a4004359585f16799f86f86d1707f"} Oct 29 23:46:01.993986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:01.993959 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:46:02.085001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.084963 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-plugins-config-volume\") pod \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\" (UID: \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\") " Oct 29 23:46:02.085374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.085346 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-kube-api-access-mvnsk\" (UniqueName: \"kubernetes.io/projected/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-kube-api-access-mvnsk\") pod \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\" (UID: \"3cb9bb9b-227a-41ad-8026-3ff61b4cc361\") " Oct 29 23:46:02.085724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.085692 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-plugins-config-volume" pod "3cb9bb9b-227a-41ad-8026-3ff61b4cc361" (UID: "3cb9bb9b-227a-41ad-8026-3ff61b4cc361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:02.086872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.086846 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:02.088115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.088087 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-kube-api-access-mvnsk" pod "3cb9bb9b-227a-41ad-8026-3ff61b4cc361" (UID: "3cb9bb9b-227a-41ad-8026-3ff61b4cc361"). InnerVolumeSpecName "kube-api-access-mvnsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:02.188291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.188207 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvnsk\" (UniqueName: \"kubernetes.io/projected/3cb9bb9b-227a-41ad-8026-3ff61b4cc361-kube-api-access-mvnsk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:02.928910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.928868 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" event={"ID":"3cb9bb9b-227a-41ad-8026-3ff61b4cc361","Type":"ContainerDied","Data":"ddb551bff68e9761fd3b7f2655ea525d97d8c97268fcbff8f9b0d23cb3a239f4"} Oct 29 23:46:02.929408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.928933 5736 scope.go:122] "RemoveContainer" containerID="607353011d7df5696d9a8b811779bde9788a4004359585f16799f86f86d1707f" Oct 29 23:46:02.929408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.928892 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx" Oct 29 23:46:02.988284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.988248 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx"] Oct 29 23:46:02.993565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:02.993537 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c83b50-5d48cb754d-cmjcx"] Oct 29 23:46:03.008479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:03.008445 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb9bb9b-227a-41ad-8026-3ff61b4cc361" path="/var/lib/kubelet/pods/3cb9bb9b-227a-41ad-8026-3ff61b4cc361/volumes" Oct 29 23:46:03.663967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:03.663921 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd"] Oct 29 23:46:06.219998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:06.219961 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:46:06.232393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:06.232366 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:46:07.009667 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:07.009637 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:46:08.664719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:08.664667 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" podUID="d8e21279-18ed-4bc8-9214-f93a3c416a36" containerName="payload-processing" containerID="cri-o://df52c9d7919fda9ce85014e2f2059de10ebb07719fade8e455fc3aa4dcab6bed" gracePeriod=25 Oct 29 23:46:11.600633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:11.600575 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" podUID="d8e21279-18ed-4bc8-9214-f93a3c416a36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:46:13.034821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.034782 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" containerID="cri-o://78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d" gracePeriod=45 Oct 29 23:46:13.316239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.316207 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 23:46:13.408598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.405496 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-model-cache\") pod \"7d017043-51f9-4ade-9f6b-fa48f7066093\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " Oct 29 23:46:13.408598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.405574 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d017043-51f9-4ade-9f6b-fa48f7066093-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d017043-51f9-4ade-9f6b-fa48f7066093-tls-certs\") pod \"7d017043-51f9-4ade-9f6b-fa48f7066093\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " Oct 29 23:46:13.408598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.405633 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-home\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-home\") pod \"7d017043-51f9-4ade-9f6b-fa48f7066093\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " Oct 29 23:46:13.408598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.405697 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-dshm\") pod \"7d017043-51f9-4ade-9f6b-fa48f7066093\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " Oct 29 23:46:13.408598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.405943 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d017043-51f9-4ade-9f6b-fa48f7066093-kube-api-access-nnzm7\" (UniqueName: \"kubernetes.io/projected/7d017043-51f9-4ade-9f6b-fa48f7066093-kube-api-access-nnzm7\") pod \"7d017043-51f9-4ade-9f6b-fa48f7066093\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " Oct 29 23:46:13.408598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.406056 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-tmp-dir\") pod \"7d017043-51f9-4ade-9f6b-fa48f7066093\" (UID: \"7d017043-51f9-4ade-9f6b-fa48f7066093\") " Oct 29 23:46:13.408598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.408312 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-model-cache" pod "7d017043-51f9-4ade-9f6b-fa48f7066093" (UID: "7d017043-51f9-4ade-9f6b-fa48f7066093"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:13.416978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.408636 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-home" pod "7d017043-51f9-4ade-9f6b-fa48f7066093" (UID: "7d017043-51f9-4ade-9f6b-fa48f7066093"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:13.416978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.410273 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-tmp-dir" pod "7d017043-51f9-4ade-9f6b-fa48f7066093" (UID: "7d017043-51f9-4ade-9f6b-fa48f7066093"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:13.435044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.433047 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d017043-51f9-4ade-9f6b-fa48f7066093-kube-api-access-nnzm7" pod "7d017043-51f9-4ade-9f6b-fa48f7066093" (UID: "7d017043-51f9-4ade-9f6b-fa48f7066093"). InnerVolumeSpecName "kube-api-access-nnzm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:13.435044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.433742 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-dshm" pod "7d017043-51f9-4ade-9f6b-fa48f7066093" (UID: "7d017043-51f9-4ade-9f6b-fa48f7066093"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:13.435044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.433837 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d017043-51f9-4ade-9f6b-fa48f7066093-tls-certs" pod "7d017043-51f9-4ade-9f6b-fa48f7066093" (UID: "7d017043-51f9-4ade-9f6b-fa48f7066093"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:46:13.509413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.509371 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnzm7\" (UniqueName: \"kubernetes.io/projected/7d017043-51f9-4ade-9f6b-fa48f7066093-kube-api-access-nnzm7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:13.509566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.509437 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:13.509566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.509455 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:13.509566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.509468 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d017043-51f9-4ade-9f6b-fa48f7066093-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:13.509566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.509481 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:13.509566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.509494 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d017043-51f9-4ade-9f6b-fa48f7066093-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:13.904303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.904258 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9"] Oct 29 23:46:13.905329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.905297 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3cb9bb9b-227a-41ad-8026-3ff61b4cc361" containerName="payload-processing" Oct 29 23:46:13.905329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.905326 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb9bb9b-227a-41ad-8026-3ff61b4cc361" containerName="payload-processing" Oct 29 23:46:13.905329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.905338 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" Oct 29 23:46:13.905660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.905343 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" Oct 29 23:46:13.905660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.905369 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f0996c41-d830-47b5-8e47-d5e41ea27717" containerName="revoke-keys" Oct 29 23:46:13.905660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.905375 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0996c41-d830-47b5-8e47-d5e41ea27717" containerName="revoke-keys" Oct 29 23:46:13.905660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.905535 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb9bb9b-227a-41ad-8026-3ff61b4cc361" containerName="payload-processing" Oct 29 23:46:13.905660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.905551 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerName="main" Oct 29 23:46:13.905660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.905567 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0996c41-d830-47b5-8e47-d5e41ea27717" containerName="revoke-keys" Oct 29 23:46:13.910021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.909992 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:13.949645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.943137 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9"] Oct 29 23:46:13.949645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.949280 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x"] Oct 29 23:46:13.965784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.965750 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:13.973136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:13.973103 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x"] Oct 29 23:46:14.017985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.017947 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.018152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.018004 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.018152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.018120 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.018272 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.018187 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.018344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.018322 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.018458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.018443 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.018520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.018502 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.018683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.018663 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xr6\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-kube-api-access-k8xr6\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.018782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.018743 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.058356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.058323 5736 generic.go:356] "Generic (PLEG): container finished" podID="7d017043-51f9-4ade-9f6b-fa48f7066093" containerID="78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d" exitCode=0 Oct 29 23:46:14.058782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.058385 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" Oct 29 23:46:14.058782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.058421 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" event={"ID":"7d017043-51f9-4ade-9f6b-fa48f7066093","Type":"ContainerDied","Data":"78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d"} Oct 29 23:46:14.058782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.058468 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56" event={"ID":"7d017043-51f9-4ade-9f6b-fa48f7066093","Type":"ContainerDied","Data":"600979e338400fe9a19f09b84bd7e9a451e23af473d8a0b981ae3a7c398ba833"} Oct 29 23:46:14.058782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.058488 5736 scope.go:122] "RemoveContainer" containerID="78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d" Oct 29 23:46:14.070345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.070326 5736 scope.go:122] "RemoveContainer" containerID="78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d" Oct 29 23:46:14.070655 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:46:14.070631 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d\": container with ID starting with 78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d not found: ID does not exist" containerID="78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d" Oct 29 23:46:14.070725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.070663 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d"} err="failed to get container status \"78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d\": rpc error: code = NotFound desc = could not find container \"78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d\": container with ID starting with 78b22c50cf0d427176e46c34c0c80c8b50127f5a6b454a15c0815f3ccd78152d not found: ID does not exist" Oct 29 23:46:14.111346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.111320 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56"] Oct 29 23:46:14.115563 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.115535 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/simulator-w2-7aeaad-kserve-64d79ddd64-gpv56"] Oct 29 23:46:14.120326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.120301 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.120446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.120366 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.120514 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.120499 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r95c8\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-kube-api-access-r95c8\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.120695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.120663 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.120826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.120743 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.120909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.120867 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.120909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.120896 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.121060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121035 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.121138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121120 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.121217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121161 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.121283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121216 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xr6\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-kube-api-access-k8xr6\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.121330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121218 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.121330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121292 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.121409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121337 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.121409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121390 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.121517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121485 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.121593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121562 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.121640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121621 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.121697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121677 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.121849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121816 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.121968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.121944 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.122054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.122032 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.122096 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.122056 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.123075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.123051 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.123336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.123316 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.144268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.144238 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.144346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.144302 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xr6\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-kube-api-access-k8xr6\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.223440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.223396 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.223627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.223531 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.223627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.223563 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r95c8\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-kube-api-access-r95c8\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.223762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.223732 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.223852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.223836 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.223914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.223865 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.223914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.223877 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.224027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.223908 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.224087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.224021 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.224087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.224068 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.224247 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.224226 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.224362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.224302 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.224483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.224462 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.224696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.224677 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.225899 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.225872 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:14.225899 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.225884 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.226062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.226040 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.247298 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.247269 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.248081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.248058 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r95c8\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-kube-api-access-r95c8\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.285237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.285199 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:14.560000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.559962 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9"] Oct 29 23:46:14.561038 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:46:14.561012 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ea5a9d_905f_45e0_87c0_1625cb7c2618.slice/crio-674ecd409db9756004a9a9ff28f6f69fe569b8fa05642e58c1f8cfef94810fdb WatchSource:0}: Error finding container 674ecd409db9756004a9a9ff28f6f69fe569b8fa05642e58c1f8cfef94810fdb: Status 404 returned error can't find the container with id 674ecd409db9756004a9a9ff28f6f69fe569b8fa05642e58c1f8cfef94810fdb Oct 29 23:46:14.564226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.564191 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:46:14.564317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.564267 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:46:14.564317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.564297 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:46:14.628923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.628891 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x"] Oct 29 23:46:14.630046 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:46:14.630020 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bc7bb9_500f_49c8_a4dd_8011592e7de2.slice/crio-1aa6fa5716356cb09d3fdc33795bebc9e5f1c1a30bd40459597b9a0505f841a6 WatchSource:0}: Error finding container 1aa6fa5716356cb09d3fdc33795bebc9e5f1c1a30bd40459597b9a0505f841a6: Status 404 returned error can't find the container with id 1aa6fa5716356cb09d3fdc33795bebc9e5f1c1a30bd40459597b9a0505f841a6 Oct 29 23:46:14.633253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.633221 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:46:14.633306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.633281 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:46:14.633347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:14.633312 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:46:15.008859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.008824 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d017043-51f9-4ade-9f6b-fa48f7066093" path="/var/lib/kubelet/pods/7d017043-51f9-4ade-9f6b-fa48f7066093/volumes" Oct 29 23:46:15.074985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.074944 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" event={"ID":"96ea5a9d-905f-45e0-87c0-1625cb7c2618","Type":"ContainerStarted","Data":"a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd"} Oct 29 23:46:15.075480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.075054 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" event={"ID":"96ea5a9d-905f-45e0-87c0-1625cb7c2618","Type":"ContainerStarted","Data":"674ecd409db9756004a9a9ff28f6f69fe569b8fa05642e58c1f8cfef94810fdb"} Oct 29 23:46:15.076981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.076949 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" event={"ID":"d2bc7bb9-500f-49c8-a4dd-8011592e7de2","Type":"ContainerStarted","Data":"8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0"} Oct 29 23:46:15.077078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.077006 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" event={"ID":"d2bc7bb9-500f-49c8-a4dd-8011592e7de2","Type":"ContainerStarted","Data":"1aa6fa5716356cb09d3fdc33795bebc9e5f1c1a30bd40459597b9a0505f841a6"} Oct 29 23:46:15.126600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.124838 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" podStartSLOduration=2.124819392 podStartE2EDuration="2.124819392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:15.119436901 +0000 UTC m=+3772.235356509" watchObservedRunningTime="2026-10-29 23:46:15.124819392 +0000 UTC m=+3772.240738999" Oct 29 23:46:15.169719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.169648 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" podStartSLOduration=2.169628939 podStartE2EDuration="2.169628939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:15.166749768 +0000 UTC m=+3772.282669378" watchObservedRunningTime="2026-10-29 23:46:15.169628939 +0000 UTC m=+3772.285548545" Oct 29 23:46:15.226165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.226127 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:15.227769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.227734 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" start-of-body= Oct 29 23:46:15.227893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.227791 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" Oct 29 23:46:15.285533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.285447 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:15.287111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.287079 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" start-of-body= Oct 29 23:46:15.287235 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:15.287140 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" Oct 29 23:46:16.226198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:16.226156 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" start-of-body= Oct 29 23:46:16.226625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:16.226221 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" Oct 29 23:46:16.288090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:16.288045 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" start-of-body= Oct 29 23:46:16.288303 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:16.288117 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" Oct 29 23:46:17.226976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:17.226944 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" start-of-body= Oct 29 23:46:17.227422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:17.227396 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" Oct 29 23:46:17.285708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:17.285655 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" start-of-body= Oct 29 23:46:17.285923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:17.285726 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" Oct 29 23:46:18.226687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:18.226646 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" start-of-body= Oct 29 23:46:18.226889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:18.226702 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" Oct 29 23:46:18.286152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:18.286107 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" start-of-body= Oct 29 23:46:18.286556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:18.286172 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" Oct 29 23:46:19.226786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:19.226744 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" start-of-body= Oct 29 23:46:19.226967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:19.226794 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" Oct 29 23:46:19.286337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:19.286279 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" start-of-body= Oct 29 23:46:19.286872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:19.286350 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" Oct 29 23:46:20.226909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:20.226871 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" start-of-body= Oct 29 23:46:20.227122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:20.226933 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.162:15021/healthz/ready\": dial tcp 10.134.1.162:15021: connect: connection refused" Oct 29 23:46:20.285616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:20.285542 5736 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" start-of-body= Oct 29 23:46:20.285799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:20.285633 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.163:15021/healthz/ready\": dial tcp 10.134.1.163:15021: connect: connection refused" Oct 29 23:46:20.709607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:20.704386 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd"] Oct 29 23:46:20.709607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:20.704691 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" containerID="cri-o://ef97e707d4f40077988eb36feb135293cd9677b217edd67da13073d2ac750538" gracePeriod=30 Oct 29 23:46:21.269971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:21.269934 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:21.271414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:21.271389 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:21.273341 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:21.273315 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:46:21.358320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:21.358280 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:21.359109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:21.359078 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:21.360273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:21.360242 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:46:21.600631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:21.600504 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" podUID="d8e21279-18ed-4bc8-9214-f93a3c416a36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:46:26.529573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:26.527508 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh"] Oct 29 23:46:26.529573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:26.527808 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" podUID="5de3e4a7-4aae-4765-b89b-8795c01ac9d2" containerName="maas-api" containerID="cri-o://c6302f4ef4d26ea5e51d7beb48dfb7167acae0419c4f0edc1a04cac91676bf6c" gracePeriod=30 Oct 29 23:46:26.567037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:26.567000 5736 generic.go:356] "Generic (PLEG): container finished" podID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerID="ef97e707d4f40077988eb36feb135293cd9677b217edd67da13073d2ac750538" exitCode=0 Oct 29 23:46:26.567477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:26.567159 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" event={"ID":"ec4f22e2-6ee6-4b14-a672-e44a57fb5270","Type":"ContainerDied","Data":"ef97e707d4f40077988eb36feb135293cd9677b217edd67da13073d2ac750538"} Oct 29 23:46:26.855526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:26.855443 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:46:27.048605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.046151 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" podUID="5de3e4a7-4aae-4765-b89b-8795c01ac9d2" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.92:8443/health\": dial tcp 10.134.1.92:8443: connect: connection refused" Oct 29 23:46:27.071603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.056765 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-envoy\") pod \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " Oct 29 23:46:27.101717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.101678 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-socket\") pod \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " Oct 29 23:46:27.120220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.120183 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istiod-ca-cert\") pod \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " Oct 29 23:46:27.120516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.120491 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-podinfo\") pod \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " Oct 29 23:46:27.120708 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.120690 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-token\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-token\") pod \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " Oct 29 23:46:27.120852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.120837 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-certs\") pod \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " Oct 29 23:46:27.121399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.121371 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-kube-api-access-bblbd\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-kube-api-access-bblbd\") pod \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " Oct 29 23:46:27.121792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.121766 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-credential-socket\") pod \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " Oct 29 23:46:27.122005 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.121986 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-data\") pod \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\" (UID: \"ec4f22e2-6ee6-4b14-a672-e44a57fb5270\") " Oct 29 23:46:27.123453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.123419 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-envoy" pod "ec4f22e2-6ee6-4b14-a672-e44a57fb5270" (UID: "ec4f22e2-6ee6-4b14-a672-e44a57fb5270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:27.142953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.142899 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-socket" pod "ec4f22e2-6ee6-4b14-a672-e44a57fb5270" (UID: "ec4f22e2-6ee6-4b14-a672-e44a57fb5270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:27.143741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.143708 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istiod-ca-cert" pod "ec4f22e2-6ee6-4b14-a672-e44a57fb5270" (UID: "ec4f22e2-6ee6-4b14-a672-e44a57fb5270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:27.155566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.155525 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-podinfo" pod "ec4f22e2-6ee6-4b14-a672-e44a57fb5270" (UID: "ec4f22e2-6ee6-4b14-a672-e44a57fb5270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:46:27.160236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.159191 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-credential-socket" pod "ec4f22e2-6ee6-4b14-a672-e44a57fb5270" (UID: "ec4f22e2-6ee6-4b14-a672-e44a57fb5270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:27.160236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.159232 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-data" pod "ec4f22e2-6ee6-4b14-a672-e44a57fb5270" (UID: "ec4f22e2-6ee6-4b14-a672-e44a57fb5270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:27.160638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.160602 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-certs" pod "ec4f22e2-6ee6-4b14-a672-e44a57fb5270" (UID: "ec4f22e2-6ee6-4b14-a672-e44a57fb5270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:27.177849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.177121 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-kube-api-access-bblbd" pod "ec4f22e2-6ee6-4b14-a672-e44a57fb5270" (UID: "ec4f22e2-6ee6-4b14-a672-e44a57fb5270"). InnerVolumeSpecName "kube-api-access-bblbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:27.207888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.207841 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-token" pod "ec4f22e2-6ee6-4b14-a672-e44a57fb5270" (UID: "ec4f22e2-6ee6-4b14-a672-e44a57fb5270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:27.264653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.234207 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:27.264971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.264941 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:27.270723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.270687 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:27.293606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.277817 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:27.293901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.293877 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:27.294064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.294051 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:27.294186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.294172 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bblbd\" (UniqueName: \"kubernetes.io/projected/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-kube-api-access-bblbd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:27.294300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.294287 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:27.294453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.294439 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4f22e2-6ee6-4b14-a672-e44a57fb5270-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:27.294557 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:46:27.262021 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:46:27.710724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.710685 5736 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:46:27.756995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.756892 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h"] Oct 29 23:46:27.799396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.799353 5736 generic.go:356] "Generic (PLEG): container finished" podID="5de3e4a7-4aae-4765-b89b-8795c01ac9d2" containerID="c6302f4ef4d26ea5e51d7beb48dfb7167acae0419c4f0edc1a04cac91676bf6c" exitCode=0 Oct 29 23:46:27.800063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.800030 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" event={"ID":"5de3e4a7-4aae-4765-b89b-8795c01ac9d2","Type":"ContainerDied","Data":"c6302f4ef4d26ea5e51d7beb48dfb7167acae0419c4f0edc1a04cac91676bf6c"} Oct 29 23:46:27.802910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.802883 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" event={"ID":"ec4f22e2-6ee6-4b14-a672-e44a57fb5270","Type":"ContainerDied","Data":"82667c909b8e8ca5fd7276909657347c67440f278c6fb2fd96f49cf7ed5ad69c"} Oct 29 23:46:27.803119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.803101 5736 scope.go:122] "RemoveContainer" containerID="ef97e707d4f40077988eb36feb135293cd9677b217edd67da13073d2ac750538" Oct 29 23:46:27.803434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.803413 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd" Oct 29 23:46:27.834807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.820695 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51462d-z5z4h"] Oct 29 23:46:27.917973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.917938 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd"] Oct 29 23:46:27.925410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.925347 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-qdhnd"] Oct 29 23:46:27.962886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.961846 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5"] Oct 29 23:46:27.962886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:27.962091 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" containerID="cri-o://4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47" gracePeriod=30 Oct 29 23:46:28.014011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.013931 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:46:28.079627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.078112 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-kube-api-access-qz9b5\" (UniqueName: \"kubernetes.io/projected/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-kube-api-access-qz9b5\") pod \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " Oct 29 23:46:28.079627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.078250 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-openshift-service-ca\") pod \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " Oct 29 23:46:28.079627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.078378 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-maas-api-tls\") pod \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " Oct 29 23:46:28.079627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.078541 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-metrics-certs\") pod \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\" (UID: \"5de3e4a7-4aae-4765-b89b-8795c01ac9d2\") " Oct 29 23:46:28.102602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.090537 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-openshift-service-ca" pod "5de3e4a7-4aae-4765-b89b-8795c01ac9d2" (UID: "5de3e4a7-4aae-4765-b89b-8795c01ac9d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:28.102602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.096067 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-kube-api-access-qz9b5" pod "5de3e4a7-4aae-4765-b89b-8795c01ac9d2" (UID: "5de3e4a7-4aae-4765-b89b-8795c01ac9d2"). InnerVolumeSpecName "kube-api-access-qz9b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:28.102602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.098086 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-maas-api-tls" pod "5de3e4a7-4aae-4765-b89b-8795c01ac9d2" (UID: "5de3e4a7-4aae-4765-b89b-8795c01ac9d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:46:28.102602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.099313 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-metrics-certs" pod "5de3e4a7-4aae-4765-b89b-8795c01ac9d2" (UID: "5de3e4a7-4aae-4765-b89b-8795c01ac9d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:46:28.193056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.193007 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:28.193545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.193518 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:28.193886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.193850 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:28.194146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.194129 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz9b5\" (UniqueName: \"kubernetes.io/projected/5de3e4a7-4aae-4765-b89b-8795c01ac9d2-kube-api-access-qz9b5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:28.816430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.816391 5736 generic.go:356] "Generic (PLEG): container finished" podID="d8e21279-18ed-4bc8-9214-f93a3c416a36" containerID="df52c9d7919fda9ce85014e2f2059de10ebb07719fade8e455fc3aa4dcab6bed" exitCode=0 Oct 29 23:46:28.816989 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.816463 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" event={"ID":"d8e21279-18ed-4bc8-9214-f93a3c416a36","Type":"ContainerDied","Data":"df52c9d7919fda9ce85014e2f2059de10ebb07719fade8e455fc3aa4dcab6bed"} Oct 29 23:46:28.817996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.817971 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" event={"ID":"5de3e4a7-4aae-4765-b89b-8795c01ac9d2","Type":"ContainerDied","Data":"5d6abbbb05b7847676c024a56c8b711465b5260e562e917707fda9577c5ea945"} Oct 29 23:46:28.818121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.818016 5736 scope.go:122] "RemoveContainer" containerID="c6302f4ef4d26ea5e51d7beb48dfb7167acae0419c4f0edc1a04cac91676bf6c" Oct 29 23:46:28.818121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.818108 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh" Oct 29 23:46:28.877155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.877123 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh"] Oct 29 23:46:28.895622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.893831 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51462d-f4fb84cbd-pr9gh"] Oct 29 23:46:28.962902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:28.962873 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 23:46:29.009853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.009814 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e21279-18ed-4bc8-9214-f93a3c416a36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e21279-18ed-4bc8-9214-f93a3c416a36-plugins-config-volume\") pod \"d8e21279-18ed-4bc8-9214-f93a3c416a36\" (UID: \"d8e21279-18ed-4bc8-9214-f93a3c416a36\") " Oct 29 23:46:29.010058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.010000 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e21279-18ed-4bc8-9214-f93a3c416a36-kube-api-access-5qg77\" (UniqueName: \"kubernetes.io/projected/d8e21279-18ed-4bc8-9214-f93a3c416a36-kube-api-access-5qg77\") pod \"d8e21279-18ed-4bc8-9214-f93a3c416a36\" (UID: \"d8e21279-18ed-4bc8-9214-f93a3c416a36\") " Oct 29 23:46:29.012678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.012643 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e21279-18ed-4bc8-9214-f93a3c416a36-plugins-config-volume" pod "d8e21279-18ed-4bc8-9214-f93a3c416a36" (UID: "d8e21279-18ed-4bc8-9214-f93a3c416a36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:29.013359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.013335 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e21279-18ed-4bc8-9214-f93a3c416a36-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:29.017111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.016449 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e21279-18ed-4bc8-9214-f93a3c416a36-kube-api-access-5qg77" pod "d8e21279-18ed-4bc8-9214-f93a3c416a36" (UID: "d8e21279-18ed-4bc8-9214-f93a3c416a36"). InnerVolumeSpecName "kube-api-access-5qg77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:29.017347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.017318 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de3e4a7-4aae-4765-b89b-8795c01ac9d2" path="/var/lib/kubelet/pods/5de3e4a7-4aae-4765-b89b-8795c01ac9d2/volumes" Oct 29 23:46:29.018370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.018352 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b94d6d5-7d12-4c1e-9a2d-060b12d6e338" path="/var/lib/kubelet/pods/9b94d6d5-7d12-4c1e-9a2d-060b12d6e338/volumes" Oct 29 23:46:29.019195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.019175 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" path="/var/lib/kubelet/pods/ec4f22e2-6ee6-4b14-a672-e44a57fb5270/volumes" Oct 29 23:46:29.114713 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.114674 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qg77\" (UniqueName: \"kubernetes.io/projected/d8e21279-18ed-4bc8-9214-f93a3c416a36-kube-api-access-5qg77\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:29.829001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.828959 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" event={"ID":"d8e21279-18ed-4bc8-9214-f93a3c416a36","Type":"ContainerDied","Data":"db2aa42ce93462a5a9c29fa2fa778a37adc79230e50fca74cb0553094b799ff1"} Oct 29 23:46:29.829433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.829054 5736 scope.go:122] "RemoveContainer" containerID="df52c9d7919fda9ce85014e2f2059de10ebb07719fade8e455fc3aa4dcab6bed" Oct 29 23:46:29.829433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.828975 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd" Oct 29 23:46:29.884045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.884008 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd"] Oct 29 23:46:29.895508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:29.895472 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7aeaad-6d9d6ffd56-glhhd"] Oct 29 23:46:30.939952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:30.939915 5736 scope.go:122] "RemoveContainer" containerID="a879deccc3f4d296ac49fb7ad77ad42beda41bf56866fb63c4bde7ec6a76eeb5" Oct 29 23:46:30.953750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:30.953724 5736 scope.go:122] "RemoveContainer" containerID="56e1aacd6bd94c6993ac35f3b87c086dc044ca46782ca34ea94d3918991008e6" Oct 29 23:46:31.016349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:31.016315 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e21279-18ed-4bc8-9214-f93a3c416a36" path="/var/lib/kubelet/pods/d8e21279-18ed-4bc8-9214-f93a3c416a36/volumes" Oct 29 23:46:33.027755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.027720 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh"] Oct 29 23:46:33.040212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.040178 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6f162a-w7xsh"] Oct 29 23:46:33.670897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.670861 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:46:33.792385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792299 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-envoy\") pod \"ed043325-0e1b-4c86-89dc-fee8978ff145\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " Oct 29 23:46:33.792385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792365 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-token\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-token\") pod \"ed043325-0e1b-4c86-89dc-fee8978ff145\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " Oct 29 23:46:33.792660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792390 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed043325-0e1b-4c86-89dc-fee8978ff145-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed043325-0e1b-4c86-89dc-fee8978ff145-istiod-ca-cert\") pod \"ed043325-0e1b-4c86-89dc-fee8978ff145\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " Oct 29 23:46:33.792660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792407 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-data\") pod \"ed043325-0e1b-4c86-89dc-fee8978ff145\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " Oct 29 23:46:33.792660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792487 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-credential-socket\") pod \"ed043325-0e1b-4c86-89dc-fee8978ff145\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " Oct 29 23:46:33.792660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792534 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-podinfo\") pod \"ed043325-0e1b-4c86-89dc-fee8978ff145\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " Oct 29 23:46:33.792660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792656 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-certs\") pod \"ed043325-0e1b-4c86-89dc-fee8978ff145\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " Oct 29 23:46:33.792980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792834 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-data" pod "ed043325-0e1b-4c86-89dc-fee8978ff145" (UID: "ed043325-0e1b-4c86-89dc-fee8978ff145"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:33.792980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792853 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-credential-socket" pod "ed043325-0e1b-4c86-89dc-fee8978ff145" (UID: "ed043325-0e1b-4c86-89dc-fee8978ff145"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:33.792980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792877 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed043325-0e1b-4c86-89dc-fee8978ff145-istiod-ca-cert" pod "ed043325-0e1b-4c86-89dc-fee8978ff145" (UID: "ed043325-0e1b-4c86-89dc-fee8978ff145"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:33.792980 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792899 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-socket\") pod \"ed043325-0e1b-4c86-89dc-fee8978ff145\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " Oct 29 23:46:33.793218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.792999 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-kube-api-access-lhswp\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-kube-api-access-lhswp\") pod \"ed043325-0e1b-4c86-89dc-fee8978ff145\" (UID: \"ed043325-0e1b-4c86-89dc-fee8978ff145\") " Oct 29 23:46:33.793218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.793120 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-certs" pod "ed043325-0e1b-4c86-89dc-fee8978ff145" (UID: "ed043325-0e1b-4c86-89dc-fee8978ff145"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:33.793406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.793226 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-socket" pod "ed043325-0e1b-4c86-89dc-fee8978ff145" (UID: "ed043325-0e1b-4c86-89dc-fee8978ff145"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:33.793727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.793705 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:33.793727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.793727 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed043325-0e1b-4c86-89dc-fee8978ff145-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:33.793896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.793739 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:33.793896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.793748 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:33.793896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.793756 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:33.794753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.794729 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-token" pod "ed043325-0e1b-4c86-89dc-fee8978ff145" (UID: "ed043325-0e1b-4c86-89dc-fee8978ff145"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:33.794877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.794861 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-podinfo" pod "ed043325-0e1b-4c86-89dc-fee8978ff145" (UID: "ed043325-0e1b-4c86-89dc-fee8978ff145"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:46:33.795011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.794997 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-envoy" pod "ed043325-0e1b-4c86-89dc-fee8978ff145" (UID: "ed043325-0e1b-4c86-89dc-fee8978ff145"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:33.795302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.795286 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-kube-api-access-lhswp" pod "ed043325-0e1b-4c86-89dc-fee8978ff145" (UID: "ed043325-0e1b-4c86-89dc-fee8978ff145"). InnerVolumeSpecName "kube-api-access-lhswp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:33.869449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.869359 5736 generic.go:356] "Generic (PLEG): container finished" podID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerID="4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47" exitCode=0 Oct 29 23:46:33.869449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.869410 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" Oct 29 23:46:33.869718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.869448 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" event={"ID":"ed043325-0e1b-4c86-89dc-fee8978ff145","Type":"ContainerDied","Data":"4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47"} Oct 29 23:46:33.869718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.869544 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5" event={"ID":"ed043325-0e1b-4c86-89dc-fee8978ff145","Type":"ContainerDied","Data":"b1e9526f72abf69b0d875adb39a0ba156c5d909cb163284a61ec50c30453b734"} Oct 29 23:46:33.869718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.869575 5736 scope.go:122] "RemoveContainer" containerID="4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47" Oct 29 23:46:33.882459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.882436 5736 scope.go:122] "RemoveContainer" containerID="4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47" Oct 29 23:46:33.882764 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:46:33.882742 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47\": container with ID starting with 4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47 not found: ID does not exist" containerID="4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47" Oct 29 23:46:33.882842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.882772 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47"} err="failed to get container status \"4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47\": rpc error: code = NotFound desc = could not find container \"4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47\": container with ID starting with 4c193b6c11a7ef6f31e486f7acd1327c0ebe15e52e447c05d470cccac8088c47 not found: ID does not exist" Oct 29 23:46:33.895841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.895816 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:33.895841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.895844 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:33.896049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.895855 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed043325-0e1b-4c86-89dc-fee8978ff145-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:33.896049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.895865 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhswp\" (UniqueName: \"kubernetes.io/projected/ed043325-0e1b-4c86-89dc-fee8978ff145-kube-api-access-lhswp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:33.948124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.948091 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5"] Oct 29 23:46:33.960769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:33.960737 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-slrm5"] Oct 29 23:46:35.012905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:35.012871 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d66738-1738-4538-9efb-31db79c154e8" path="/var/lib/kubelet/pods/41d66738-1738-4538-9efb-31db79c154e8/volumes" Oct 29 23:46:35.013502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:35.013485 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" path="/var/lib/kubelet/pods/ed043325-0e1b-4c86-89dc-fee8978ff145/volumes" Oct 29 23:46:37.099037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.098990 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8"] Oct 29 23:46:37.099464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.099337 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" containerID="cri-o://17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f" gracePeriod=30 Oct 29 23:46:37.731245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.731198 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h"] Oct 29 23:46:37.732079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732032 5736 cpu_manager.go:414] "containerMap: removing container" podUID="5de3e4a7-4aae-4765-b89b-8795c01ac9d2" containerName="maas-api" Oct 29 23:46:37.732079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732058 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de3e4a7-4aae-4765-b89b-8795c01ac9d2" containerName="maas-api" Oct 29 23:46:37.732287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732108 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d8e21279-18ed-4bc8-9214-f93a3c416a36" containerName="payload-processing" Oct 29 23:46:37.732287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732117 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e21279-18ed-4bc8-9214-f93a3c416a36" containerName="payload-processing" Oct 29 23:46:37.732287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732151 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" Oct 29 23:46:37.732287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732160 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" Oct 29 23:46:37.732287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732182 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" Oct 29 23:46:37.732287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732188 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" Oct 29 23:46:37.732612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732368 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de3e4a7-4aae-4765-b89b-8795c01ac9d2" containerName="maas-api" Oct 29 23:46:37.732612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732392 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e21279-18ed-4bc8-9214-f93a3c416a36" containerName="payload-processing" Oct 29 23:46:37.732612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732406 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4f22e2-6ee6-4b14-a672-e44a57fb5270" containerName="istio-proxy" Oct 29 23:46:37.732612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.732420 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed043325-0e1b-4c86-89dc-fee8978ff145" containerName="istio-proxy" Oct 29 23:46:37.737967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.737945 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:37.763795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.763763 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:46:37.770721 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.770685 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h"] Oct 29 23:46:37.861221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.861185 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:46:37.886434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.886395 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tk9\" (UniqueName: \"kubernetes.io/projected/fb421517-b825-439d-952c-67956c876066-kube-api-access-w8tk9\") pod \"maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:37.886626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.886589 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb421517-b825-439d-952c-67956c876066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:37.886687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.886657 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb421517-b825-439d-952c-67956c876066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:37.987943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.987861 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb421517-b825-439d-952c-67956c876066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:37.987943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.987923 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb421517-b825-439d-952c-67956c876066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:37.988137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.988125 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tk9\" (UniqueName: \"kubernetes.io/projected/fb421517-b825-439d-952c-67956c876066-kube-api-access-w8tk9\") pod \"maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:37.988647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.988620 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb421517-b825-439d-952c-67956c876066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:37.990511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:37.990483 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb421517-b825-439d-952c-67956c876066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:38.012138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:38.012108 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tk9\" (UniqueName: \"kubernetes.io/projected/fb421517-b825-439d-952c-67956c876066-kube-api-access-w8tk9\") pod \"maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:38.054049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:38.054013 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:38.372189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:38.372152 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h"] Oct 29 23:46:38.374441 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:46:38.374407 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb421517_b825_439d_952c_67956c876066.slice/crio-d878c35bfcc9bb7eedf3c2b27c35e30033ecfc26424bade0f641a1b1e3aead0f WatchSource:0}: Error finding container d878c35bfcc9bb7eedf3c2b27c35e30033ecfc26424bade0f641a1b1e3aead0f: Status 404 returned error can't find the container with id d878c35bfcc9bb7eedf3c2b27c35e30033ecfc26424bade0f641a1b1e3aead0f Oct 29 23:46:38.917951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:38.917911 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" event={"ID":"fb421517-b825-439d-952c-67956c876066","Type":"ContainerStarted","Data":"5df5d90dd2a6f6979736dc62df07c20bbc1a7d410cd7582380aecfe105f12d06"} Oct 29 23:46:38.918130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:38.917978 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" event={"ID":"fb421517-b825-439d-952c-67956c876066","Type":"ContainerStarted","Data":"d878c35bfcc9bb7eedf3c2b27c35e30033ecfc26424bade0f641a1b1e3aead0f"} Oct 29 23:46:38.961909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:38.961848 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" podStartSLOduration=1.961831431 podStartE2EDuration="1.961831431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:38.958485083 +0000 UTC m=+3796.074404688" watchObservedRunningTime="2026-10-29 23:46:38.961831431 +0000 UTC m=+3796.077751038" Oct 29 23:46:39.927207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:39.927170 5736 generic.go:356] "Generic (PLEG): container finished" podID="fb421517-b825-439d-952c-67956c876066" containerID="5df5d90dd2a6f6979736dc62df07c20bbc1a7d410cd7582380aecfe105f12d06" exitCode=0 Oct 29 23:46:39.927606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:39.927260 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" event={"ID":"fb421517-b825-439d-952c-67956c876066","Type":"ContainerDied","Data":"5df5d90dd2a6f6979736dc62df07c20bbc1a7d410cd7582380aecfe105f12d06"} Oct 29 23:46:40.879674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:40.879638 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4"] Oct 29 23:46:40.884059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:40.884039 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:40.906245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:40.906220 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4"] Oct 29 23:46:40.951333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:40.951289 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zb4h\" (UniqueName: \"kubernetes.io/projected/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-kube-api-access-6zb4h\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:40.951815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:40.951723 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:40.951898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:40.951821 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:41.053234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.053187 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zb4h\" (UniqueName: \"kubernetes.io/projected/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-kube-api-access-6zb4h\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:41.053768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.053740 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:41.053894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.053807 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:41.054637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.054610 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:41.056413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.056369 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:41.076821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.076800 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zb4h\" (UniqueName: \"kubernetes.io/projected/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-kube-api-access-6zb4h\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:41.123954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.123929 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:41.155132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.155073 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb421517-b825-439d-952c-67956c876066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb421517-b825-439d-952c-67956c876066-maas-api-xks-ca\") pod \"fb421517-b825-439d-952c-67956c876066\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " Oct 29 23:46:41.155228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.155133 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb421517-b825-439d-952c-67956c876066-kube-api-access-w8tk9\" (UniqueName: \"kubernetes.io/projected/fb421517-b825-439d-952c-67956c876066-kube-api-access-w8tk9\") pod \"fb421517-b825-439d-952c-67956c876066\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " Oct 29 23:46:41.155319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.155302 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb421517-b825-439d-952c-67956c876066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb421517-b825-439d-952c-67956c876066-maas-api-openshift-ca\") pod \"fb421517-b825-439d-952c-67956c876066\" (UID: \"fb421517-b825-439d-952c-67956c876066\") " Oct 29 23:46:41.155687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.155661 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb421517-b825-439d-952c-67956c876066-maas-api-openshift-ca" pod "fb421517-b825-439d-952c-67956c876066" (UID: "fb421517-b825-439d-952c-67956c876066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:41.156384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.156356 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb421517-b825-439d-952c-67956c876066-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:41.157186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.157147 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb421517-b825-439d-952c-67956c876066-maas-api-xks-ca" pod "fb421517-b825-439d-952c-67956c876066" (UID: "fb421517-b825-439d-952c-67956c876066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:46:41.157421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.157404 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb421517-b825-439d-952c-67956c876066-kube-api-access-w8tk9" pod "fb421517-b825-439d-952c-67956c876066" (UID: "fb421517-b825-439d-952c-67956c876066"). InnerVolumeSpecName "kube-api-access-w8tk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:41.198549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.198521 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:41.258111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.258072 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb421517-b825-439d-952c-67956c876066-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:41.258325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.258114 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8tk9\" (UniqueName: \"kubernetes.io/projected/fb421517-b825-439d-952c-67956c876066-kube-api-access-w8tk9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:41.514193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.514164 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4"] Oct 29 23:46:41.517069 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:46:41.517040 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1574a4_ca73_4e1c_bba8_f5a6e3355875.slice/crio-008f5b20ab829d3ae63231bb7215ccfad73ecdc9d3e25bfd814a363d548806f7 WatchSource:0}: Error finding container 008f5b20ab829d3ae63231bb7215ccfad73ecdc9d3e25bfd814a363d548806f7: Status 404 returned error can't find the container with id 008f5b20ab829d3ae63231bb7215ccfad73ecdc9d3e25bfd814a363d548806f7 Oct 29 23:46:41.947533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.947500 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" event={"ID":"bb1574a4-ca73-4e1c-bba8-f5a6e3355875","Type":"ContainerStarted","Data":"ad1475236eeddd66570c2442f42ac8eb57763029ba50af86a5cacbef3274e14a"} Oct 29 23:46:41.947769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.947555 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" event={"ID":"bb1574a4-ca73-4e1c-bba8-f5a6e3355875","Type":"ContainerStarted","Data":"008f5b20ab829d3ae63231bb7215ccfad73ecdc9d3e25bfd814a363d548806f7"} Oct 29 23:46:41.949043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.949022 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" Oct 29 23:46:41.949152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.949059 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h" event={"ID":"fb421517-b825-439d-952c-67956c876066","Type":"ContainerDied","Data":"d878c35bfcc9bb7eedf3c2b27c35e30033ecfc26424bade0f641a1b1e3aead0f"} Oct 29 23:46:41.949152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.949096 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d878c35bfcc9bb7eedf3c2b27c35e30033ecfc26424bade0f641a1b1e3aead0f" Oct 29 23:46:41.989025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:41.988980 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" podStartSLOduration=1.9889669049999998 podStartE2EDuration="1.988966905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:41.987212747 +0000 UTC m=+3799.103132376" watchObservedRunningTime="2026-10-29 23:46:41.988966905 +0000 UTC m=+3799.104886510" Oct 29 23:46:42.389128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.389101 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:46:42.479759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.479725 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-token\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-token\") pod \"d1712a1a-9e91-4570-995c-20e50ade63bf\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " Oct 29 23:46:42.479916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.479795 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-credential-socket\") pod \"d1712a1a-9e91-4570-995c-20e50ade63bf\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " Oct 29 23:46:42.480009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.479936 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-socket\") pod \"d1712a1a-9e91-4570-995c-20e50ade63bf\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " Oct 29 23:46:42.480009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.479974 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-certs\") pod \"d1712a1a-9e91-4570-995c-20e50ade63bf\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " Oct 29 23:46:42.480162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.480041 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-envoy\") pod \"d1712a1a-9e91-4570-995c-20e50ade63bf\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " Oct 29 23:46:42.480162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.480154 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-kube-api-access-r6j8s\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-kube-api-access-r6j8s\") pod \"d1712a1a-9e91-4570-995c-20e50ade63bf\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " Oct 29 23:46:42.480361 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.480341 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-data\") pod \"d1712a1a-9e91-4570-995c-20e50ade63bf\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " Oct 29 23:46:42.480424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.480373 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-socket" pod "d1712a1a-9e91-4570-995c-20e50ade63bf" (UID: "d1712a1a-9e91-4570-995c-20e50ade63bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.480424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.480396 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1712a1a-9e91-4570-995c-20e50ade63bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1712a1a-9e91-4570-995c-20e50ade63bf-istiod-ca-cert\") pod \"d1712a1a-9e91-4570-995c-20e50ade63bf\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " Oct 29 23:46:42.480536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.480461 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-podinfo\") pod \"d1712a1a-9e91-4570-995c-20e50ade63bf\" (UID: \"d1712a1a-9e91-4570-995c-20e50ade63bf\") " Oct 29 23:46:42.481276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.481245 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.481691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.481668 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-data" pod "d1712a1a-9e91-4570-995c-20e50ade63bf" (UID: "d1712a1a-9e91-4570-995c-20e50ade63bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.481796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.481719 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-certs" pod "d1712a1a-9e91-4570-995c-20e50ade63bf" (UID: "d1712a1a-9e91-4570-995c-20e50ade63bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.482059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.482036 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1712a1a-9e91-4570-995c-20e50ade63bf-istiod-ca-cert" pod "d1712a1a-9e91-4570-995c-20e50ade63bf" (UID: "d1712a1a-9e91-4570-995c-20e50ade63bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:42.483179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.483148 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-credential-socket" pod "d1712a1a-9e91-4570-995c-20e50ade63bf" (UID: "d1712a1a-9e91-4570-995c-20e50ade63bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.484682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.484658 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-podinfo" pod "d1712a1a-9e91-4570-995c-20e50ade63bf" (UID: "d1712a1a-9e91-4570-995c-20e50ade63bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:46:42.484945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.484700 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-token" pod "d1712a1a-9e91-4570-995c-20e50ade63bf" (UID: "d1712a1a-9e91-4570-995c-20e50ade63bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:42.486131 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.486107 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-kube-api-access-r6j8s" pod "d1712a1a-9e91-4570-995c-20e50ade63bf" (UID: "d1712a1a-9e91-4570-995c-20e50ade63bf"). InnerVolumeSpecName "kube-api-access-r6j8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:42.486217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.486109 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-envoy" pod "d1712a1a-9e91-4570-995c-20e50ade63bf" (UID: "d1712a1a-9e91-4570-995c-20e50ade63bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:46:42.582650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.582617 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.582650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.582647 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.582856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.582658 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.582856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.582667 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.582856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.582676 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6j8s\" (UniqueName: \"kubernetes.io/projected/d1712a1a-9e91-4570-995c-20e50ade63bf-kube-api-access-r6j8s\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.582856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.582687 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.582856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.582698 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1712a1a-9e91-4570-995c-20e50ade63bf-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.582856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.582706 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1712a1a-9e91-4570-995c-20e50ade63bf-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:42.959385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.959343 5736 generic.go:356] "Generic (PLEG): container finished" podID="bb1574a4-ca73-4e1c-bba8-f5a6e3355875" containerID="ad1475236eeddd66570c2442f42ac8eb57763029ba50af86a5cacbef3274e14a" exitCode=0 Oct 29 23:46:42.959571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.959426 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" event={"ID":"bb1574a4-ca73-4e1c-bba8-f5a6e3355875","Type":"ContainerDied","Data":"ad1475236eeddd66570c2442f42ac8eb57763029ba50af86a5cacbef3274e14a"} Oct 29 23:46:42.960890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.960866 5736 generic.go:356] "Generic (PLEG): container finished" podID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerID="17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f" exitCode=0 Oct 29 23:46:42.961010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.960901 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" Oct 29 23:46:42.961010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.960928 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" event={"ID":"d1712a1a-9e91-4570-995c-20e50ade63bf","Type":"ContainerDied","Data":"17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f"} Oct 29 23:46:42.961010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.960966 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8" event={"ID":"d1712a1a-9e91-4570-995c-20e50ade63bf","Type":"ContainerDied","Data":"8d82b8f381b62857f6663a013e7ae60b172c503633bc02e9a4505482162e3b1b"} Oct 29 23:46:42.961010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.960986 5736 scope.go:122] "RemoveContainer" containerID="17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f" Oct 29 23:46:42.972998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.972981 5736 scope.go:122] "RemoveContainer" containerID="17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f" Oct 29 23:46:42.973292 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:46:42.973272 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f\": container with ID starting with 17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f not found: ID does not exist" containerID="17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f" Oct 29 23:46:42.973357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:42.973302 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f"} err="failed to get container status \"17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f\": rpc error: code = NotFound desc = could not find container \"17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f\": container with ID starting with 17a5dc5c1f3dd4b89b6da7335e8a3c3ebf62d9e7fc612a89314479f655bbd85f not found: ID does not exist" Oct 29 23:46:43.047995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:43.047841 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8"] Oct 29 23:46:43.053940 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:43.053912 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-51462d-openshift-default-5dfc96f495-7hln8"] Oct 29 23:46:44.148504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.148476 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:44.207807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.207769 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-openshift-ca\") pod \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " Oct 29 23:46:44.208039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.208020 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-xks-ca\") pod \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " Oct 29 23:46:44.208168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.208153 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-kube-api-access-6zb4h\" (UniqueName: \"kubernetes.io/projected/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-kube-api-access-6zb4h\") pod \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\" (UID: \"bb1574a4-ca73-4e1c-bba8-f5a6e3355875\") " Oct 29 23:46:44.208237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.208196 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-openshift-ca" pod "bb1574a4-ca73-4e1c-bba8-f5a6e3355875" (UID: "bb1574a4-ca73-4e1c-bba8-f5a6e3355875"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:46:44.208999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.208968 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:44.210548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.210509 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-xks-ca" pod "bb1574a4-ca73-4e1c-bba8-f5a6e3355875" (UID: "bb1574a4-ca73-4e1c-bba8-f5a6e3355875"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:46:44.210788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.210761 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-kube-api-access-6zb4h" pod "bb1574a4-ca73-4e1c-bba8-f5a6e3355875" (UID: "bb1574a4-ca73-4e1c-bba8-f5a6e3355875"). InnerVolumeSpecName "kube-api-access-6zb4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:46:44.310533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.310492 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zb4h\" (UniqueName: \"kubernetes.io/projected/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-kube-api-access-6zb4h\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:44.310533 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.310528 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1574a4-ca73-4e1c-bba8-f5a6e3355875-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:46:44.981354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.981318 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" Oct 29 23:46:44.981525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.981352 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4" event={"ID":"bb1574a4-ca73-4e1c-bba8-f5a6e3355875","Type":"ContainerDied","Data":"008f5b20ab829d3ae63231bb7215ccfad73ecdc9d3e25bfd814a363d548806f7"} Oct 29 23:46:44.981525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:44.981427 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="008f5b20ab829d3ae63231bb7215ccfad73ecdc9d3e25bfd814a363d548806f7" Oct 29 23:46:45.006342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:45.006315 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" path="/var/lib/kubelet/pods/d1712a1a-9e91-4570-995c-20e50ade63bf/volumes" Oct 29 23:46:54.505875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:54.505839 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch"] Oct 29 23:46:54.517189 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:54.517152 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7d4110-s7bch"] Oct 29 23:46:55.006340 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:46:55.006305 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26dfcc8-e552-4811-b74e-74bd5631dd7a" path="/var/lib/kubelet/pods/c26dfcc8-e552-4811-b74e-74bd5631dd7a/volumes" Oct 29 23:46:56.825759 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:46:56.825716 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:47:01.685425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:01.685387 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh"] Oct 29 23:47:01.686373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:01.686318 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" containerID="cri-o://b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487" gracePeriod=30 Oct 29 23:47:02.345696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.345658 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w"] Oct 29 23:47:02.346372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.346354 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fb421517-b825-439d-952c-67956c876066" containerName="revoke-keys" Oct 29 23:47:02.346449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.346374 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb421517-b825-439d-952c-67956c876066" containerName="revoke-keys" Oct 29 23:47:02.346449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.346430 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" Oct 29 23:47:02.346449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.346436 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" Oct 29 23:47:02.346559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.346451 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bb1574a4-ca73-4e1c-bba8-f5a6e3355875" containerName="revoke-keys" Oct 29 23:47:02.346559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.346457 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1574a4-ca73-4e1c-bba8-f5a6e3355875" containerName="revoke-keys" Oct 29 23:47:02.346646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.346564 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1712a1a-9e91-4570-995c-20e50ade63bf" containerName="istio-proxy" Oct 29 23:47:02.346646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.346574 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1574a4-ca73-4e1c-bba8-f5a6e3355875" containerName="revoke-keys" Oct 29 23:47:02.346646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.346596 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb421517-b825-439d-952c-67956c876066" containerName="revoke-keys" Oct 29 23:47:02.350481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.350457 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.379741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.379707 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w"] Oct 29 23:47:02.409972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.409936 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:47:02.410144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.410022 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:47:02.467494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.467456 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.467709 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.467589 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.467765 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.467740 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zj2\" (UniqueName: \"kubernetes.io/projected/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-kube-api-access-m4zj2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.569656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.569618 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.569849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.569697 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.569849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.569777 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zj2\" (UniqueName: \"kubernetes.io/projected/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-kube-api-access-m4zj2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.570388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.570364 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.572008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.571989 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.593695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.593660 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zj2\" (UniqueName: \"kubernetes.io/projected/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-kube-api-access-m4zj2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:02.679838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:02.679286 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:03.004988 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:47:03.003901 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489cc3d7_538f_4d0d_a9bc_3fc787a3ffaf.slice/crio-92fbe966f4abbdc979d33699073955fe279724e61ed7ff8ab990dfa1330fc93f WatchSource:0}: Error finding container 92fbe966f4abbdc979d33699073955fe279724e61ed7ff8ab990dfa1330fc93f: Status 404 returned error can't find the container with id 92fbe966f4abbdc979d33699073955fe279724e61ed7ff8ab990dfa1330fc93f Oct 29 23:47:03.010291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.010262 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w"] Oct 29 23:47:03.272082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.271999 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" event={"ID":"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf","Type":"ContainerStarted","Data":"92fbe966f4abbdc979d33699073955fe279724e61ed7ff8ab990dfa1330fc93f"} Oct 29 23:47:03.392419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.392381 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj"] Oct 29 23:47:03.399887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.399856 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.416160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.416128 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-37498a-openshift-default-dockercfg-pjlnk\"" Oct 29 23:47:03.432562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.432529 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj"] Oct 29 23:47:03.490055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.490013 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-certs\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.490392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.490144 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05cf00-34b7-4564-b736-950698149d88-istio-podinfo\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.490392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.490178 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05cf00-34b7-4564-b736-950698149d88-istiod-ca-cert\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.490392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.490326 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-credential-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.490566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.490442 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dkv\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-kube-api-access-d7dkv\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.490691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.490611 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-data\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.490691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.490648 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-envoy\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.490691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.490668 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.490691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.490690 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-istio-token\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.591963 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-certs\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592110 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05cf00-34b7-4564-b736-950698149d88-istio-podinfo\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592141 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05cf00-34b7-4564-b736-950698149d88-istiod-ca-cert\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592247 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-credential-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592356 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592333 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-certs\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592381 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dkv\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-kube-api-access-d7dkv\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592503 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-data\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592552 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-envoy\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592678 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-credential-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592716 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592760 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-istio-token\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.592894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592866 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-data\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.593014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.592995 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05cf00-34b7-4564-b736-950698149d88-istiod-ca-cert\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.593088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.593045 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-socket\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.594711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.594690 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-envoy\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.594932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.594913 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05cf00-34b7-4564-b736-950698149d88-istio-podinfo\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.615592 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.615552 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dkv\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-kube-api-access-d7dkv\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.615693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.615568 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-istio-token\") pod \"e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:03.722427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:03.722390 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:04.062604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.062563 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj"] Oct 29 23:47:04.065132 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:47:04.065093 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce05cf00_34b7_4564_b736_950698149d88.slice/crio-99fe55f1c76ead6461463a21f725825f9a0e06c3d3de769f25a488c820906e72 WatchSource:0}: Error finding container 99fe55f1c76ead6461463a21f725825f9a0e06c3d3de769f25a488c820906e72: Status 404 returned error can't find the container with id 99fe55f1c76ead6461463a21f725825f9a0e06c3d3de769f25a488c820906e72 Oct 29 23:47:04.068622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.068551 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:47:04.068734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.068658 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:47:04.068734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.068692 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:47:04.282263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.282227 5736 generic.go:356] "Generic (PLEG): container finished" podID="489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf" containerID="5ee3281c41ac411ee6ac2c7964f1393b80d888d38c4ec7c2474ad9f325cf9a77" exitCode=0 Oct 29 23:47:04.282456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.282322 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" event={"ID":"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf","Type":"ContainerDied","Data":"5ee3281c41ac411ee6ac2c7964f1393b80d888d38c4ec7c2474ad9f325cf9a77"} Oct 29 23:47:04.284420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.284396 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" event={"ID":"ce05cf00-34b7-4564-b736-950698149d88","Type":"ContainerStarted","Data":"70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb"} Oct 29 23:47:04.284546 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.284440 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" event={"ID":"ce05cf00-34b7-4564-b736-950698149d88","Type":"ContainerStarted","Data":"99fe55f1c76ead6461463a21f725825f9a0e06c3d3de769f25a488c820906e72"} Oct 29 23:47:04.369692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.369551 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" podStartSLOduration=1.369504623 podStartE2EDuration="1.369504623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:47:04.367080563 +0000 UTC m=+3821.483000169" watchObservedRunningTime="2026-10-29 23:47:04.369504623 +0000 UTC m=+3821.485424229" Oct 29 23:47:04.723706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:04.723650 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:05.505309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.505278 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:05.655655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.644143 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-kube-api-access-m4zj2\" (UniqueName: \"kubernetes.io/projected/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-kube-api-access-m4zj2\") pod \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " Oct 29 23:47:05.655655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.644212 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-xks-ca\") pod \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " Oct 29 23:47:05.655655 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.644538 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-openshift-ca\") pod \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\" (UID: \"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf\") " Oct 29 23:47:05.655921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.655807 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-openshift-ca" pod "489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf" (UID: "489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:05.658035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.658007 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-xks-ca" pod "489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf" (UID: "489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:47:05.658188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.658067 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-kube-api-access-m4zj2" pod "489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf" (UID: "489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf"). InnerVolumeSpecName "kube-api-access-m4zj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:05.724198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.724166 5736 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" start-of-body= Oct 29 23:47:05.724359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.724221 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" podUID="ce05cf00-34b7-4564-b736-950698149d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" Oct 29 23:47:05.756128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.753029 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:05.756128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.755659 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4zj2\" (UniqueName: \"kubernetes.io/projected/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-kube-api-access-m4zj2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:05.756128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:05.755692 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:06.304551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:06.304514 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" Oct 29 23:47:06.304751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:06.304546 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w" event={"ID":"489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf","Type":"ContainerDied","Data":"92fbe966f4abbdc979d33699073955fe279724e61ed7ff8ab990dfa1330fc93f"} Oct 29 23:47:06.304751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:06.304613 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92fbe966f4abbdc979d33699073955fe279724e61ed7ff8ab990dfa1330fc93f" Oct 29 23:47:06.723673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:06.723638 5736 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" start-of-body= Oct 29 23:47:06.723997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:06.723702 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" podUID="ce05cf00-34b7-4564-b736-950698149d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" Oct 29 23:47:07.022913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.022881 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:47:07.182531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.182494 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-socket\") pod \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " Oct 29 23:47:07.182746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.182626 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-podinfo\") pod \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " Oct 29 23:47:07.182746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.182683 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-credential-socket\") pod \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " Oct 29 23:47:07.182746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.182712 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-token\") pod \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " Oct 29 23:47:07.182881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.182754 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-socket" pod "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" (UID: "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:07.182881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.182839 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-envoy\") pod \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " Oct 29 23:47:07.182881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.182868 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-kube-api-access-9wrp4\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-kube-api-access-9wrp4\") pod \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " Oct 29 23:47:07.183055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.182921 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-certs\") pod \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " Oct 29 23:47:07.183055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.182979 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-credential-socket" pod "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" (UID: "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:07.183181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.183118 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istiod-ca-cert\") pod \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " Oct 29 23:47:07.183253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.183182 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-data\") pod \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\" (UID: \"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac\") " Oct 29 23:47:07.183347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.183321 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-certs" pod "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" (UID: "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:07.183671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.183641 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-data" pod "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" (UID: "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:07.183972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.183947 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:07.183972 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.183971 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:07.184149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.183980 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:07.184149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.183989 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:07.184149 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.183963 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istiod-ca-cert" pod "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" (UID: "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:07.185339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.185307 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-kube-api-access-9wrp4" pod "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" (UID: "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac"). InnerVolumeSpecName "kube-api-access-9wrp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:07.185339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.185329 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-podinfo" pod "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" (UID: "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:47:07.185494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.185422 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-token" pod "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" (UID: "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:07.185494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.185489 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-envoy" pod "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" (UID: "a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:07.286177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.286089 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:07.286177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.286125 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wrp4\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-kube-api-access-9wrp4\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:07.286177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.286137 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:07.286177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.286150 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:07.286177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.286159 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:07.314433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.314400 5736 generic.go:356] "Generic (PLEG): container finished" podID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerID="b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487" exitCode=0 Oct 29 23:47:07.314610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.314458 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" Oct 29 23:47:07.314610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.314488 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" event={"ID":"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac","Type":"ContainerDied","Data":"b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487"} Oct 29 23:47:07.314610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.314558 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh" event={"ID":"a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac","Type":"ContainerDied","Data":"35f12b713dde5e7c6b64bc82fa7c69859d1049f2581e24fb74058014c4084bbb"} Oct 29 23:47:07.314881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.314620 5736 scope.go:122] "RemoveContainer" containerID="b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487" Oct 29 23:47:07.326915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.326882 5736 scope.go:122] "RemoveContainer" containerID="37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8" Oct 29 23:47:07.366091 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.366068 5736 scope.go:122] "RemoveContainer" containerID="b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487" Oct 29 23:47:07.366417 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:47:07.366395 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487\": container with ID starting with b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487 not found: ID does not exist" containerID="b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487" Oct 29 23:47:07.366495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.366430 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487"} err="failed to get container status \"b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487\": rpc error: code = NotFound desc = could not find container \"b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487\": container with ID starting with b47d24eaad08a0a8cad9efb2d638134575a9e4ebcd4fec0653480c95828ad487 not found: ID does not exist" Oct 29 23:47:07.366495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.366447 5736 scope.go:122] "RemoveContainer" containerID="37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8" Oct 29 23:47:07.366760 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:47:07.366742 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8\": container with ID starting with 37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8 not found: ID does not exist" containerID="37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8" Oct 29 23:47:07.366819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.366765 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8"} err="failed to get container status \"37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8\": rpc error: code = NotFound desc = could not find container \"37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8\": container with ID starting with 37599059e2834978e3f72b7268723e5a1902c83574e83a0ebccfb11160e63fd8 not found: ID does not exist" Oct 29 23:47:07.375494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.375469 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh"] Oct 29 23:47:07.378999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.378977 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7d4110-openshift-default-5bbb7f6bd9-krvjh"] Oct 29 23:47:07.723063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.723029 5736 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" start-of-body= Oct 29 23:47:07.723242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:07.723086 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" podUID="ce05cf00-34b7-4564-b736-950698149d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" Oct 29 23:47:08.722868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:08.722828 5736 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" start-of-body= Oct 29 23:47:08.723368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:08.722890 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" podUID="ce05cf00-34b7-4564-b736-950698149d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" Oct 29 23:47:09.006986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.006911 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" path="/var/lib/kubelet/pods/a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac/volumes" Oct 29 23:47:09.723781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.723744 5736 patch_prober.go:29] interesting pod/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" start-of-body= Oct 29 23:47:09.724267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.723799 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" podUID="ce05cf00-34b7-4564-b736-950698149d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.167:15021/healthz/ready\": dial tcp 10.134.1.167:15021: connect: connection refused" Oct 29 23:47:09.977207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.977125 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc"] Oct 29 23:47:09.979257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.979092 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" Oct 29 23:47:09.979257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.979124 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" Oct 29 23:47:09.979257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.979211 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" Oct 29 23:47:09.979257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.979227 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" Oct 29 23:47:09.979595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.979306 5736 cpu_manager.go:414] "containerMap: removing container" podUID="489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf" containerName="revoke-keys" Oct 29 23:47:09.979595 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.979316 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf" containerName="revoke-keys" Oct 29 23:47:09.979718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.979668 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" Oct 29 23:47:09.979718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.979705 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf" containerName="revoke-keys" Oct 29 23:47:09.986529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.986506 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:09.997404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.997226 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx"] Oct 29 23:47:09.998267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:09.998242 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ee0caa-a5e0-4c18-81c5-0e675a4ba9ac" containerName="istio-proxy" Oct 29 23:47:10.002141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.002116 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.005220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.005195 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc"] Oct 29 23:47:10.005826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.005802 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:47:10.005926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.005879 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:47:10.027381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.027350 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx"] Oct 29 23:47:10.029481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.029428 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmb6g\" (UniqueName: \"kubernetes.io/projected/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-kube-api-access-qmb6g\") pod \"maas-api-revoke-sub-13952795-xpmsc\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.029844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.029822 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf15c06-ftntx\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.030051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.030031 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf15c06-ftntx\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.030207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.030188 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13952795-xpmsc\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.030377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.030343 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcr58\" (UniqueName: \"kubernetes.io/projected/84ba6468-9ab8-4864-b79f-512acb32e038-kube-api-access-bcr58\") pod \"maas-api-revoke-sub-7cf15c06-ftntx\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.030537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.030519 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13952795-xpmsc\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.148266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.148219 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcr58\" (UniqueName: \"kubernetes.io/projected/84ba6468-9ab8-4864-b79f-512acb32e038-kube-api-access-bcr58\") pod \"maas-api-revoke-sub-7cf15c06-ftntx\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.148466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.148322 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13952795-xpmsc\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.148664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.148639 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmb6g\" (UniqueName: \"kubernetes.io/projected/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-kube-api-access-qmb6g\") pod \"maas-api-revoke-sub-13952795-xpmsc\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.148774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.148756 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf15c06-ftntx\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.148859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.148817 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf15c06-ftntx\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.148923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.148869 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13952795-xpmsc\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.149130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.149104 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13952795-xpmsc\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.149674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.149644 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf15c06-ftntx\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.151749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.151714 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf15c06-ftntx\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.151932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.151910 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13952795-xpmsc\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.172252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.172218 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmb6g\" (UniqueName: \"kubernetes.io/projected/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-kube-api-access-qmb6g\") pod \"maas-api-revoke-sub-13952795-xpmsc\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.172252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.172242 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcr58\" (UniqueName: \"kubernetes.io/projected/84ba6468-9ab8-4864-b79f-512acb32e038-kube-api-access-bcr58\") pod \"maas-api-revoke-sub-7cf15c06-ftntx\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.306184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.306094 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:10.323808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.323781 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:10.643841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.643804 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc"] Oct 29 23:47:10.644446 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:47:10.644418 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f25381_b24f_4bdd_ab20_33f64c0d9ea2.slice/crio-a5057792cecf64ac2623d0ff81548a0991a96c038d053121e8d5d319abebfc9d WatchSource:0}: Error finding container a5057792cecf64ac2623d0ff81548a0991a96c038d053121e8d5d319abebfc9d: Status 404 returned error can't find the container with id a5057792cecf64ac2623d0ff81548a0991a96c038d053121e8d5d319abebfc9d Oct 29 23:47:10.657874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.657843 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx"] Oct 29 23:47:10.661706 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:47:10.661680 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ba6468_9ab8_4864_b79f_512acb32e038.slice/crio-2912be6911e3bf0f6501fb8a440dad6da56a58cdf2402cd688e0f454fe0d7c33 WatchSource:0}: Error finding container 2912be6911e3bf0f6501fb8a440dad6da56a58cdf2402cd688e0f454fe0d7c33: Status 404 returned error can't find the container with id 2912be6911e3bf0f6501fb8a440dad6da56a58cdf2402cd688e0f454fe0d7c33 Oct 29 23:47:10.756484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.756436 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:10.757444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.757373 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:10.758345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:10.758319 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:47:11.366284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:11.366249 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" event={"ID":"84ba6468-9ab8-4864-b79f-512acb32e038","Type":"ContainerStarted","Data":"fc2b6f959da4de0aff0b3cfe7a810cbde1a48939bb2eab82f27a180adb7c6334"} Oct 29 23:47:11.366470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:11.366305 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" event={"ID":"84ba6468-9ab8-4864-b79f-512acb32e038","Type":"ContainerStarted","Data":"2912be6911e3bf0f6501fb8a440dad6da56a58cdf2402cd688e0f454fe0d7c33"} Oct 29 23:47:11.367968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:11.367941 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" event={"ID":"64f25381-b24f-4bdd-ab20-33f64c0d9ea2","Type":"ContainerStarted","Data":"8214f45212d049926b1464d56e8c008abe0524af863559a0d036e17ded0484e4"} Oct 29 23:47:11.368085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:11.367990 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" event={"ID":"64f25381-b24f-4bdd-ab20-33f64c0d9ea2","Type":"ContainerStarted","Data":"a5057792cecf64ac2623d0ff81548a0991a96c038d053121e8d5d319abebfc9d"} Oct 29 23:47:11.408598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:11.407411 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" podStartSLOduration=2.407398497 podStartE2EDuration="2.407398497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:47:11.406518971 +0000 UTC m=+3828.522438578" watchObservedRunningTime="2026-10-29 23:47:11.407398497 +0000 UTC m=+3828.523318103" Oct 29 23:47:11.448362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:11.448306 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" podStartSLOduration=2.448289646 podStartE2EDuration="2.448289646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:47:11.445933433 +0000 UTC m=+3828.561853039" watchObservedRunningTime="2026-10-29 23:47:11.448289646 +0000 UTC m=+3828.564209253" Oct 29 23:47:12.387496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:12.387432 5736 generic.go:356] "Generic (PLEG): container finished" podID="84ba6468-9ab8-4864-b79f-512acb32e038" containerID="fc2b6f959da4de0aff0b3cfe7a810cbde1a48939bb2eab82f27a180adb7c6334" exitCode=0 Oct 29 23:47:12.388774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:12.387558 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" event={"ID":"84ba6468-9ab8-4864-b79f-512acb32e038","Type":"ContainerDied","Data":"fc2b6f959da4de0aff0b3cfe7a810cbde1a48939bb2eab82f27a180adb7c6334"} Oct 29 23:47:12.391081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:12.391053 5736 generic.go:356] "Generic (PLEG): container finished" podID="64f25381-b24f-4bdd-ab20-33f64c0d9ea2" containerID="8214f45212d049926b1464d56e8c008abe0524af863559a0d036e17ded0484e4" exitCode=0 Oct 29 23:47:12.391319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:12.391131 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" event={"ID":"64f25381-b24f-4bdd-ab20-33f64c0d9ea2","Type":"ContainerDied","Data":"8214f45212d049926b1464d56e8c008abe0524af863559a0d036e17ded0484e4"} Oct 29 23:47:13.889969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.889936 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:13.892487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.892469 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:13.972001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.971966 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-xks-ca\") pod \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " Oct 29 23:47:13.972168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.972134 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-openshift-ca\") pod \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " Oct 29 23:47:13.972255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.972189 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-openshift-ca\") pod \"84ba6468-9ab8-4864-b79f-512acb32e038\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " Oct 29 23:47:13.972255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.972239 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ba6468-9ab8-4864-b79f-512acb32e038-kube-api-access-bcr58\" (UniqueName: \"kubernetes.io/projected/84ba6468-9ab8-4864-b79f-512acb32e038-kube-api-access-bcr58\") pod \"84ba6468-9ab8-4864-b79f-512acb32e038\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " Oct 29 23:47:13.972387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.972309 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-xks-ca\") pod \"84ba6468-9ab8-4864-b79f-512acb32e038\" (UID: \"84ba6468-9ab8-4864-b79f-512acb32e038\") " Oct 29 23:47:13.972548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.972525 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-kube-api-access-qmb6g\" (UniqueName: \"kubernetes.io/projected/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-kube-api-access-qmb6g\") pod \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\" (UID: \"64f25381-b24f-4bdd-ab20-33f64c0d9ea2\") " Oct 29 23:47:13.972732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.972697 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-openshift-ca" pod "84ba6468-9ab8-4864-b79f-512acb32e038" (UID: "84ba6468-9ab8-4864-b79f-512acb32e038"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:13.973207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.973165 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-openshift-ca" pod "64f25381-b24f-4bdd-ab20-33f64c0d9ea2" (UID: "64f25381-b24f-4bdd-ab20-33f64c0d9ea2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:13.973473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.973445 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:13.973629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.973481 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:13.975434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.974681 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-xks-ca" pod "64f25381-b24f-4bdd-ab20-33f64c0d9ea2" (UID: "64f25381-b24f-4bdd-ab20-33f64c0d9ea2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:47:13.977956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.977490 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-kube-api-access-qmb6g" pod "64f25381-b24f-4bdd-ab20-33f64c0d9ea2" (UID: "64f25381-b24f-4bdd-ab20-33f64c0d9ea2"). InnerVolumeSpecName "kube-api-access-qmb6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:13.978070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.978041 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-xks-ca" pod "84ba6468-9ab8-4864-b79f-512acb32e038" (UID: "84ba6468-9ab8-4864-b79f-512acb32e038"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:47:13.981744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:13.981061 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ba6468-9ab8-4864-b79f-512acb32e038-kube-api-access-bcr58" pod "84ba6468-9ab8-4864-b79f-512acb32e038" (UID: "84ba6468-9ab8-4864-b79f-512acb32e038"). InnerVolumeSpecName "kube-api-access-bcr58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:14.074699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.074646 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcr58\" (UniqueName: \"kubernetes.io/projected/84ba6468-9ab8-4864-b79f-512acb32e038-kube-api-access-bcr58\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:14.074699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.074685 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ba6468-9ab8-4864-b79f-512acb32e038-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:14.074699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.074702 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmb6g\" (UniqueName: \"kubernetes.io/projected/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-kube-api-access-qmb6g\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:14.074957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.074717 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f25381-b24f-4bdd-ab20-33f64c0d9ea2-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:14.649887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.649853 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" Oct 29 23:47:14.650086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.649881 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx" event={"ID":"84ba6468-9ab8-4864-b79f-512acb32e038","Type":"ContainerDied","Data":"2912be6911e3bf0f6501fb8a440dad6da56a58cdf2402cd688e0f454fe0d7c33"} Oct 29 23:47:14.650086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.649932 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2912be6911e3bf0f6501fb8a440dad6da56a58cdf2402cd688e0f454fe0d7c33" Oct 29 23:47:14.651448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.651425 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" Oct 29 23:47:14.651448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.651437 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc" event={"ID":"64f25381-b24f-4bdd-ab20-33f64c0d9ea2","Type":"ContainerDied","Data":"a5057792cecf64ac2623d0ff81548a0991a96c038d053121e8d5d319abebfc9d"} Oct 29 23:47:14.651641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:14.651472 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5057792cecf64ac2623d0ff81548a0991a96c038d053121e8d5d319abebfc9d" Oct 29 23:47:26.867630 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:47:26.863771 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:47:31.066790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:31.066753 5736 scope.go:122] "RemoveContainer" containerID="7fddcdfdbe9d1aea84cb1f849cce2712555521d696700cbc872c4466fed23076" Oct 29 23:47:44.473985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:44.473943 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4"] Oct 29 23:47:44.485389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:44.485341 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb367d-pzsr4"] Oct 29 23:47:45.005114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:45.005078 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1574a4-ca73-4e1c-bba8-f5a6e3355875" path="/var/lib/kubelet/pods/bb1574a4-ca73-4e1c-bba8-f5a6e3355875/volumes" Oct 29 23:47:47.114110 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:47.112639 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h"] Oct 29 23:47:47.117034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:47.116994 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b7a567-8bp4h"] Oct 29 23:47:49.006562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:49.006527 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb421517-b825-439d-952c-67956c876066" path="/var/lib/kubelet/pods/fb421517-b825-439d-952c-67956c876066/volumes" Oct 29 23:47:51.602767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:51.602728 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k"] Oct 29 23:47:51.603483 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:51.603450 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" containerID="cri-o://6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc" gracePeriod=30 Oct 29 23:47:52.231575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.231539 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf"] Oct 29 23:47:52.232248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.232232 5736 cpu_manager.go:414] "containerMap: removing container" podUID="64f25381-b24f-4bdd-ab20-33f64c0d9ea2" containerName="revoke-keys" Oct 29 23:47:52.232292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.232249 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f25381-b24f-4bdd-ab20-33f64c0d9ea2" containerName="revoke-keys" Oct 29 23:47:52.232292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.232285 5736 cpu_manager.go:414] "containerMap: removing container" podUID="84ba6468-9ab8-4864-b79f-512acb32e038" containerName="revoke-keys" Oct 29 23:47:52.232292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.232291 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ba6468-9ab8-4864-b79f-512acb32e038" containerName="revoke-keys" Oct 29 23:47:52.232407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.232398 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ba6468-9ab8-4864-b79f-512acb32e038" containerName="revoke-keys" Oct 29 23:47:52.232445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.232419 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f25381-b24f-4bdd-ab20-33f64c0d9ea2" containerName="revoke-keys" Oct 29 23:47:52.236331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.236312 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.257746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.257721 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf"] Oct 29 23:47:52.297688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.297654 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.297847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.297745 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.297897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.297847 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qkv\" (UniqueName: \"kubernetes.io/projected/4bc35f7b-ab42-4434-ac06-740df27fdc35-kube-api-access-68qkv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.303785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.303757 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:47:52.304030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.304010 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:47:52.399424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.399372 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.399646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.399480 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.399646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.399513 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qkv\" (UniqueName: \"kubernetes.io/projected/4bc35f7b-ab42-4434-ac06-740df27fdc35-kube-api-access-68qkv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.400816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.400788 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.412689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.412658 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.422957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.422931 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qkv\" (UniqueName: \"kubernetes.io/projected/4bc35f7b-ab42-4434-ac06-740df27fdc35-kube-api-access-68qkv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.550307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.550235 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:52.884421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:52.884381 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf"] Oct 29 23:47:52.885639 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:47:52.885609 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc35f7b_ab42_4434_ac06_740df27fdc35.slice/crio-d3267d2218e0dfbacbcf3c6d571abd4997a27c8288d379463550ca70acfad534 WatchSource:0}: Error finding container d3267d2218e0dfbacbcf3c6d571abd4997a27c8288d379463550ca70acfad534: Status 404 returned error can't find the container with id d3267d2218e0dfbacbcf3c6d571abd4997a27c8288d379463550ca70acfad534 Oct 29 23:47:53.216800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:53.216763 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" event={"ID":"4bc35f7b-ab42-4434-ac06-740df27fdc35","Type":"ContainerStarted","Data":"d3267d2218e0dfbacbcf3c6d571abd4997a27c8288d379463550ca70acfad534"} Oct 29 23:47:54.304753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:54.304716 5736 generic.go:356] "Generic (PLEG): container finished" podID="4bc35f7b-ab42-4434-ac06-740df27fdc35" containerID="b445121a50da7ef0504f294fffbbd666c913470a81e0898222fb93f9d27ed3fe" exitCode=0 Oct 29 23:47:54.305522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:54.305490 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" event={"ID":"4bc35f7b-ab42-4434-ac06-740df27fdc35","Type":"ContainerDied","Data":"b445121a50da7ef0504f294fffbbd666c913470a81e0898222fb93f9d27ed3fe"} Oct 29 23:47:54.469394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:54.469356 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw"] Oct 29 23:47:54.470046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:54.469849 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" containerID="cri-o://eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e" gracePeriod=30 Oct 29 23:47:55.077759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.077713 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j"] Oct 29 23:47:55.083828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.083797 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.093675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.093651 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.094003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.093974 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44phg\" (UniqueName: \"kubernetes.io/projected/9df8a42f-cade-4197-a586-e895939a9edc-kube-api-access-44phg\") pod \"maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.094458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.094433 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.105699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.105673 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j"] Oct 29 23:47:55.196000 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.195949 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44phg\" (UniqueName: \"kubernetes.io/projected/9df8a42f-cade-4197-a586-e895939a9edc-kube-api-access-44phg\") pod \"maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.196226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.196194 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.196409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.196388 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.196971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.196943 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.199473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.199446 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.219974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.219954 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44phg\" (UniqueName: \"kubernetes.io/projected/9df8a42f-cade-4197-a586-e895939a9edc-kube-api-access-44phg\") pod \"maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.404382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.402309 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:55.511625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.509733 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:55.605273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.605231 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-xks-ca\") pod \"4bc35f7b-ab42-4434-ac06-740df27fdc35\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " Oct 29 23:47:55.605846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.605298 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc35f7b-ab42-4434-ac06-740df27fdc35-kube-api-access-68qkv\" (UniqueName: \"kubernetes.io/projected/4bc35f7b-ab42-4434-ac06-740df27fdc35-kube-api-access-68qkv\") pod \"4bc35f7b-ab42-4434-ac06-740df27fdc35\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " Oct 29 23:47:55.605846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.605534 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-openshift-ca\") pod \"4bc35f7b-ab42-4434-ac06-740df27fdc35\" (UID: \"4bc35f7b-ab42-4434-ac06-740df27fdc35\") " Oct 29 23:47:55.606026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.605957 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-openshift-ca" pod "4bc35f7b-ab42-4434-ac06-740df27fdc35" (UID: "4bc35f7b-ab42-4434-ac06-740df27fdc35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:55.606451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.606423 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:55.607574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.607542 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-xks-ca" pod "4bc35f7b-ab42-4434-ac06-740df27fdc35" (UID: "4bc35f7b-ab42-4434-ac06-740df27fdc35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:47:55.607960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.607938 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc35f7b-ab42-4434-ac06-740df27fdc35-kube-api-access-68qkv" pod "4bc35f7b-ab42-4434-ac06-740df27fdc35" (UID: "4bc35f7b-ab42-4434-ac06-740df27fdc35"). InnerVolumeSpecName "kube-api-access-68qkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:55.708479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.708406 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bc35f7b-ab42-4434-ac06-740df27fdc35-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:55.708479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.708438 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qkv\" (UniqueName: \"kubernetes.io/projected/4bc35f7b-ab42-4434-ac06-740df27fdc35-kube-api-access-68qkv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:55.752333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:55.752301 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j"] Oct 29 23:47:55.753507 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:47:55.753478 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df8a42f_cade_4197_a586_e895939a9edc.slice/crio-bb2cf3587d4a80b3fcc348969f4a77f006c2405a226622c26e604e13f7cc90f5 WatchSource:0}: Error finding container bb2cf3587d4a80b3fcc348969f4a77f006c2405a226622c26e604e13f7cc90f5: Status 404 returned error can't find the container with id bb2cf3587d4a80b3fcc348969f4a77f006c2405a226622c26e604e13f7cc90f5 Oct 29 23:47:56.324572 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.324532 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" event={"ID":"9df8a42f-cade-4197-a586-e895939a9edc","Type":"ContainerStarted","Data":"94a4654718fde719613cbb75e5a6d93184eeba271a184ae6ef70e363d5b6f377"} Oct 29 23:47:56.324785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.324628 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" event={"ID":"9df8a42f-cade-4197-a586-e895939a9edc","Type":"ContainerStarted","Data":"bb2cf3587d4a80b3fcc348969f4a77f006c2405a226622c26e604e13f7cc90f5"} Oct 29 23:47:56.326132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.326109 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" Oct 29 23:47:56.326280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.326159 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf" event={"ID":"4bc35f7b-ab42-4434-ac06-740df27fdc35","Type":"ContainerDied","Data":"d3267d2218e0dfbacbcf3c6d571abd4997a27c8288d379463550ca70acfad534"} Oct 29 23:47:56.326280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.326208 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3267d2218e0dfbacbcf3c6d571abd4997a27c8288d379463550ca70acfad534" Oct 29 23:47:56.367984 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.367936 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" podStartSLOduration=1.367921355 podStartE2EDuration="1.367921355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:47:56.366050247 +0000 UTC m=+3873.481969852" watchObservedRunningTime="2026-10-29 23:47:56.367921355 +0000 UTC m=+3873.483840960" Oct 29 23:47:56.842898 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:47:56.842852 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:47:56.893895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.893835 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:47:56.928025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.927998 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-data\") pod \"bf67b3ca-b596-4145-b77b-53c1207c0efd\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " Oct 29 23:47:56.928183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928106 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-certs\") pod \"bf67b3ca-b596-4145-b77b-53c1207c0efd\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " Oct 29 23:47:56.928183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928169 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-credential-socket\") pod \"bf67b3ca-b596-4145-b77b-53c1207c0efd\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " Oct 29 23:47:56.928325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928220 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-socket\") pod \"bf67b3ca-b596-4145-b77b-53c1207c0efd\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " Oct 29 23:47:56.928325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928250 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-data" pod "bf67b3ca-b596-4145-b77b-53c1207c0efd" (UID: "bf67b3ca-b596-4145-b77b-53c1207c0efd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:56.928325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928294 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-envoy\") pod \"bf67b3ca-b596-4145-b77b-53c1207c0efd\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " Oct 29 23:47:56.928507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928341 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf67b3ca-b596-4145-b77b-53c1207c0efd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf67b3ca-b596-4145-b77b-53c1207c0efd-istiod-ca-cert\") pod \"bf67b3ca-b596-4145-b77b-53c1207c0efd\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " Oct 29 23:47:56.928507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928379 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-podinfo\") pod \"bf67b3ca-b596-4145-b77b-53c1207c0efd\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " Oct 29 23:47:56.928507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928375 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-certs" pod "bf67b3ca-b596-4145-b77b-53c1207c0efd" (UID: "bf67b3ca-b596-4145-b77b-53c1207c0efd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:56.928507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928474 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-kube-api-access-7tzb5\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-kube-api-access-7tzb5\") pod \"bf67b3ca-b596-4145-b77b-53c1207c0efd\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " Oct 29 23:47:56.928771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928485 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-credential-socket" pod "bf67b3ca-b596-4145-b77b-53c1207c0efd" (UID: "bf67b3ca-b596-4145-b77b-53c1207c0efd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:56.928771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928506 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-token\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-token\") pod \"bf67b3ca-b596-4145-b77b-53c1207c0efd\" (UID: \"bf67b3ca-b596-4145-b77b-53c1207c0efd\") " Oct 29 23:47:56.928771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928541 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-socket" pod "bf67b3ca-b596-4145-b77b-53c1207c0efd" (UID: "bf67b3ca-b596-4145-b77b-53c1207c0efd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:56.928956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.928895 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf67b3ca-b596-4145-b77b-53c1207c0efd-istiod-ca-cert" pod "bf67b3ca-b596-4145-b77b-53c1207c0efd" (UID: "bf67b3ca-b596-4145-b77b-53c1207c0efd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:56.929625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.929596 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf67b3ca-b596-4145-b77b-53c1207c0efd-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:56.929625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.929628 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:56.929831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.929641 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:56.929831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.929657 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:56.929831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.929670 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:56.930654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.930626 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-envoy" pod "bf67b3ca-b596-4145-b77b-53c1207c0efd" (UID: "bf67b3ca-b596-4145-b77b-53c1207c0efd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:47:56.930790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.930770 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-token" pod "bf67b3ca-b596-4145-b77b-53c1207c0efd" (UID: "bf67b3ca-b596-4145-b77b-53c1207c0efd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:56.930838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.930788 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-kube-api-access-7tzb5" pod "bf67b3ca-b596-4145-b77b-53c1207c0efd" (UID: "bf67b3ca-b596-4145-b77b-53c1207c0efd"). InnerVolumeSpecName "kube-api-access-7tzb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:56.930838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:56.930789 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-podinfo" pod "bf67b3ca-b596-4145-b77b-53c1207c0efd" (UID: "bf67b3ca-b596-4145-b77b-53c1207c0efd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:47:57.030673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.030633 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.030673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.030670 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.030921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.030685 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tzb5\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-kube-api-access-7tzb5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.030921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.030701 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf67b3ca-b596-4145-b77b-53c1207c0efd-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:57.336903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.336864 5736 generic.go:356] "Generic (PLEG): container finished" podID="9df8a42f-cade-4197-a586-e895939a9edc" containerID="94a4654718fde719613cbb75e5a6d93184eeba271a184ae6ef70e363d5b6f377" exitCode=0 Oct 29 23:47:57.337106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.336951 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" event={"ID":"9df8a42f-cade-4197-a586-e895939a9edc","Type":"ContainerDied","Data":"94a4654718fde719613cbb75e5a6d93184eeba271a184ae6ef70e363d5b6f377"} Oct 29 23:47:57.339418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.339390 5736 generic.go:356] "Generic (PLEG): container finished" podID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerID="6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc" exitCode=0 Oct 29 23:47:57.339601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.339458 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" Oct 29 23:47:57.339601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.339472 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" event={"ID":"bf67b3ca-b596-4145-b77b-53c1207c0efd","Type":"ContainerDied","Data":"6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc"} Oct 29 23:47:57.339601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.339531 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k" event={"ID":"bf67b3ca-b596-4145-b77b-53c1207c0efd","Type":"ContainerDied","Data":"342c10e90022ba42aedc37d111b8384848f92985a3721c0d0eb38815c31f466f"} Oct 29 23:47:57.339601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.339562 5736 scope.go:122] "RemoveContainer" containerID="6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc" Oct 29 23:47:57.351442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.351417 5736 scope.go:122] "RemoveContainer" containerID="6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc" Oct 29 23:47:57.351770 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:47:57.351747 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc\": container with ID starting with 6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc not found: ID does not exist" containerID="6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc" Oct 29 23:47:57.351879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.351779 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc"} err="failed to get container status \"6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc\": rpc error: code = NotFound desc = could not find container \"6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc\": container with ID starting with 6f88951517ac30a01a7fe15b8b9fd3f592a197243d83e80ffce13f909a0c89bc not found: ID does not exist" Oct 29 23:47:57.419814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.419778 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k"] Oct 29 23:47:57.429973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:57.429942 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb367d-openshift-default-76cbb89c4f-z9z4k"] Oct 29 23:47:58.515354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.515326 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:58.553521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.553491 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-xks-ca\") pod \"9df8a42f-cade-4197-a586-e895939a9edc\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " Oct 29 23:47:58.553747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.553731 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df8a42f-cade-4197-a586-e895939a9edc-kube-api-access-44phg\" (UniqueName: \"kubernetes.io/projected/9df8a42f-cade-4197-a586-e895939a9edc-kube-api-access-44phg\") pod \"9df8a42f-cade-4197-a586-e895939a9edc\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " Oct 29 23:47:58.553839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.553794 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-openshift-ca\") pod \"9df8a42f-cade-4197-a586-e895939a9edc\" (UID: \"9df8a42f-cade-4197-a586-e895939a9edc\") " Oct 29 23:47:58.554181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.554152 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-openshift-ca" pod "9df8a42f-cade-4197-a586-e895939a9edc" (UID: "9df8a42f-cade-4197-a586-e895939a9edc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:47:58.554486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.554466 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:58.555788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.555763 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-xks-ca" pod "9df8a42f-cade-4197-a586-e895939a9edc" (UID: "9df8a42f-cade-4197-a586-e895939a9edc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:47:58.555885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.555864 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df8a42f-cade-4197-a586-e895939a9edc-kube-api-access-44phg" pod "9df8a42f-cade-4197-a586-e895939a9edc" (UID: "9df8a42f-cade-4197-a586-e895939a9edc"). InnerVolumeSpecName "kube-api-access-44phg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:47:58.655719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.655626 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44phg\" (UniqueName: \"kubernetes.io/projected/9df8a42f-cade-4197-a586-e895939a9edc-kube-api-access-44phg\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:58.655719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:58.655666 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df8a42f-cade-4197-a586-e895939a9edc-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:47:59.006262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:59.006230 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" path="/var/lib/kubelet/pods/bf67b3ca-b596-4145-b77b-53c1207c0efd/volumes" Oct 29 23:47:59.358418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:59.358334 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" Oct 29 23:47:59.358418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:59.358368 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j" event={"ID":"9df8a42f-cade-4197-a586-e895939a9edc","Type":"ContainerDied","Data":"bb2cf3587d4a80b3fcc348969f4a77f006c2405a226622c26e604e13f7cc90f5"} Oct 29 23:47:59.358684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:47:59.358431 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb2cf3587d4a80b3fcc348969f4a77f006c2405a226622c26e604e13f7cc90f5" Oct 29 23:48:00.213334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.213305 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:48:00.281727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.281642 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171cdd2b-1e57-45e7-81db-177626f01707-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171cdd2b-1e57-45e7-81db-177626f01707-istiod-ca-cert\") pod \"171cdd2b-1e57-45e7-81db-177626f01707\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " Oct 29 23:48:00.281727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.281689 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-envoy\") pod \"171cdd2b-1e57-45e7-81db-177626f01707\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " Oct 29 23:48:00.281727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.281721 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-kube-api-access-ngsh8\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-kube-api-access-ngsh8\") pod \"171cdd2b-1e57-45e7-81db-177626f01707\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " Oct 29 23:48:00.282008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.281811 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-socket\") pod \"171cdd2b-1e57-45e7-81db-177626f01707\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " Oct 29 23:48:00.282008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.281848 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-istio-token\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-istio-token\") pod \"171cdd2b-1e57-45e7-81db-177626f01707\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " Oct 29 23:48:00.282008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.281912 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-certs\") pod \"171cdd2b-1e57-45e7-81db-177626f01707\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " Oct 29 23:48:00.282008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.281947 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-data\") pod \"171cdd2b-1e57-45e7-81db-177626f01707\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " Oct 29 23:48:00.282008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.281991 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/171cdd2b-1e57-45e7-81db-177626f01707-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171cdd2b-1e57-45e7-81db-177626f01707-istio-podinfo\") pod \"171cdd2b-1e57-45e7-81db-177626f01707\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " Oct 29 23:48:00.282284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.282084 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171cdd2b-1e57-45e7-81db-177626f01707-istiod-ca-cert" pod "171cdd2b-1e57-45e7-81db-177626f01707" (UID: "171cdd2b-1e57-45e7-81db-177626f01707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:00.282347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.282080 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-credential-socket\") pod \"171cdd2b-1e57-45e7-81db-177626f01707\" (UID: \"171cdd2b-1e57-45e7-81db-177626f01707\") " Oct 29 23:48:00.282410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.282374 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-data" pod "171cdd2b-1e57-45e7-81db-177626f01707" (UID: "171cdd2b-1e57-45e7-81db-177626f01707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:00.282472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.282401 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-socket" pod "171cdd2b-1e57-45e7-81db-177626f01707" (UID: "171cdd2b-1e57-45e7-81db-177626f01707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:00.282616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.282592 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-certs" pod "171cdd2b-1e57-45e7-81db-177626f01707" (UID: "171cdd2b-1e57-45e7-81db-177626f01707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:00.282964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.282911 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-credential-socket" pod "171cdd2b-1e57-45e7-81db-177626f01707" (UID: "171cdd2b-1e57-45e7-81db-177626f01707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:00.283394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.283357 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171cdd2b-1e57-45e7-81db-177626f01707-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.283394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.283379 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.283394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.283395 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.283794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.283406 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.283794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.283418 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.284933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.284904 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/171cdd2b-1e57-45e7-81db-177626f01707-istio-podinfo" pod "171cdd2b-1e57-45e7-81db-177626f01707" (UID: "171cdd2b-1e57-45e7-81db-177626f01707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:48:00.285156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.285127 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-istio-token" pod "171cdd2b-1e57-45e7-81db-177626f01707" (UID: "171cdd2b-1e57-45e7-81db-177626f01707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:00.285746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.285718 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-envoy" pod "171cdd2b-1e57-45e7-81db-177626f01707" (UID: "171cdd2b-1e57-45e7-81db-177626f01707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:00.287289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.287266 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-kube-api-access-ngsh8" pod "171cdd2b-1e57-45e7-81db-177626f01707" (UID: "171cdd2b-1e57-45e7-81db-177626f01707"). InnerVolumeSpecName "kube-api-access-ngsh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:00.368451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.368419 5736 generic.go:356] "Generic (PLEG): container finished" podID="171cdd2b-1e57-45e7-81db-177626f01707" containerID="eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e" exitCode=0 Oct 29 23:48:00.368675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.368472 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" Oct 29 23:48:00.368675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.368500 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" event={"ID":"171cdd2b-1e57-45e7-81db-177626f01707","Type":"ContainerDied","Data":"eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e"} Oct 29 23:48:00.368675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.368557 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw" event={"ID":"171cdd2b-1e57-45e7-81db-177626f01707","Type":"ContainerDied","Data":"6893b88d85dca2ada1fc7b92989eac5f3a54eb355bacad9c46c4d135284bd3de"} Oct 29 23:48:00.368675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.368617 5736 scope.go:122] "RemoveContainer" containerID="eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e" Oct 29 23:48:00.380894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.380876 5736 scope.go:122] "RemoveContainer" containerID="5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf" Oct 29 23:48:00.385429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.385410 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171cdd2b-1e57-45e7-81db-177626f01707-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.385541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.385432 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngsh8\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-kube-api-access-ngsh8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.385541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.385446 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171cdd2b-1e57-45e7-81db-177626f01707-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.385541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.385460 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171cdd2b-1e57-45e7-81db-177626f01707-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:00.391517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.391500 5736 scope.go:122] "RemoveContainer" containerID="eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e" Oct 29 23:48:00.391904 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:48:00.391878 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e\": container with ID starting with eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e not found: ID does not exist" containerID="eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e" Oct 29 23:48:00.391981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.391914 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e"} err="failed to get container status \"eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e\": rpc error: code = NotFound desc = could not find container \"eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e\": container with ID starting with eaab2696a7b53b07462dcb5824614e71bc591afec81501283321538403e6a51e not found: ID does not exist" Oct 29 23:48:00.391981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.391938 5736 scope.go:122] "RemoveContainer" containerID="5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf" Oct 29 23:48:00.392278 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:48:00.392253 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf\": container with ID starting with 5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf not found: ID does not exist" containerID="5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf" Oct 29 23:48:00.392362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.392282 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf"} err="failed to get container status \"5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf\": rpc error: code = NotFound desc = could not find container \"5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf\": container with ID starting with 5df924aa1571a5e7e1411037ed663344c1a5fed507258a27ba38291aae85aaaf not found: ID does not exist" Oct 29 23:48:00.436695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.436659 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw"] Oct 29 23:48:00.447637 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:00.447601 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b7a567-openshift-default-59449b8cbf-7w6nw"] Oct 29 23:48:01.005659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:01.005622 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171cdd2b-1e57-45e7-81db-177626f01707" path="/var/lib/kubelet/pods/171cdd2b-1e57-45e7-81db-177626f01707/volumes" Oct 29 23:48:06.535654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:06.529322 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w"] Oct 29 23:48:06.542221 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:06.542182 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b4b5e3-ch45w"] Oct 29 23:48:07.006393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:07.006362 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf" path="/var/lib/kubelet/pods/489cc3d7-538f-4d0d-a9bc-3fc787a3ffaf/volumes" Oct 29 23:48:13.823414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:13.823378 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6"] Oct 29 23:48:13.823906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:13.823631 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" containerID="cri-o://b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17" gracePeriod=30 Oct 29 23:48:14.044536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.044492 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz"] Oct 29 23:48:14.050239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.050208 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-328cb7e6-kprxz"] Oct 29 23:48:14.605926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.605891 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk"] Oct 29 23:48:14.607656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.607626 5736 cpu_manager.go:414] "containerMap: removing container" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" Oct 29 23:48:14.607862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.607847 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" Oct 29 23:48:14.608055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.608043 5736 cpu_manager.go:414] "containerMap: removing container" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" Oct 29 23:48:14.608166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.608154 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" Oct 29 23:48:14.631891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.631860 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4bc35f7b-ab42-4434-ac06-740df27fdc35" containerName="revoke-keys" Oct 29 23:48:14.632122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.632101 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc35f7b-ab42-4434-ac06-740df27fdc35" containerName="revoke-keys" Oct 29 23:48:14.632309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.632282 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" Oct 29 23:48:14.632426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.632413 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" Oct 29 23:48:14.632556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.632546 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9df8a42f-cade-4197-a586-e895939a9edc" containerName="revoke-keys" Oct 29 23:48:14.632672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.632658 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df8a42f-cade-4197-a586-e895939a9edc" containerName="revoke-keys" Oct 29 23:48:14.633063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.633033 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" Oct 29 23:48:14.633217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.633203 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc35f7b-ab42-4434-ac06-740df27fdc35" containerName="revoke-keys" Oct 29 23:48:14.633338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.633326 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf67b3ca-b596-4145-b77b-53c1207c0efd" containerName="istio-proxy" Oct 29 23:48:14.633455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.633444 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df8a42f-cade-4197-a586-e895939a9edc" containerName="revoke-keys" Oct 29 23:48:14.637688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.637664 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="171cdd2b-1e57-45e7-81db-177626f01707" containerName="istio-proxy" Oct 29 23:48:14.806701 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.800330 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:14.808455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.807297 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk"] Oct 29 23:48:14.824430 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.824395 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:48:14.826412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.826364 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:48:14.917707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.905287 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:14.917707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.905483 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:14.917707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:14.905603 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx2gv\" (UniqueName: \"kubernetes.io/projected/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-kube-api-access-cx2gv\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:15.010503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.008774 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:15.010503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.008899 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:15.010503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.008993 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx2gv\" (UniqueName: \"kubernetes.io/projected/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-kube-api-access-cx2gv\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:15.010503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.010314 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:15.014282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.014254 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce93aaa-0f13-4687-b455-aed1a4fce1ce" path="/var/lib/kubelet/pods/cce93aaa-0f13-4687-b455-aed1a4fce1ce/volumes" Oct 29 23:48:15.028455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.026110 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:15.050611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.048452 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx2gv\" (UniqueName: \"kubernetes.io/projected/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-kube-api-access-cx2gv\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:15.215600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.215549 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:15.385801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.385751 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j"] Oct 29 23:48:15.408640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.397689 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-dea9cb-8779j"] Oct 29 23:48:15.626656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.626496 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk"] Oct 29 23:48:15.833803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:15.833765 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" event={"ID":"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73","Type":"ContainerStarted","Data":"0d31914f5638f180e8470a615d91bd9a28800c14275b17208f9e50d09813683b"} Oct 29 23:48:16.192602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:16.165395 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr"] Oct 29 23:48:16.236982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:16.231304 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7aeaad-pjzmr"] Oct 29 23:48:16.844725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:16.844681 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" event={"ID":"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73","Type":"ContainerStarted","Data":"1c10bfb1b7ec2a16ff2e45a54d644952acfc7da7eeaf6528f0d1db425242027f"} Oct 29 23:48:16.899832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:16.899770 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" podStartSLOduration=2.899756982 podStartE2EDuration="2.899756982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:48:16.886981365 +0000 UTC m=+3894.002900985" watchObservedRunningTime="2026-10-29 23:48:16.899756982 +0000 UTC m=+3894.015676587" Oct 29 23:48:17.006068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:17.006035 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df8a42f-cade-4197-a586-e895939a9edc" path="/var/lib/kubelet/pods/9df8a42f-cade-4197-a586-e895939a9edc/volumes" Oct 29 23:48:17.006574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:17.006560 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0996c41-d830-47b5-8e47-d5e41ea27717" path="/var/lib/kubelet/pods/f0996c41-d830-47b5-8e47-d5e41ea27717/volumes" Oct 29 23:48:17.014718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:17.014693 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf"] Oct 29 23:48:17.020664 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:17.020638 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-26a2a4-45xjf"] Oct 29 23:48:17.855693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:17.855653 5736 generic.go:356] "Generic (PLEG): container finished" podID="f5bb76b8-0afa-4e1f-a888-4fb5397d8c73" containerID="1c10bfb1b7ec2a16ff2e45a54d644952acfc7da7eeaf6528f0d1db425242027f" exitCode=0 Oct 29 23:48:17.856076 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:17.855694 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" event={"ID":"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73","Type":"ContainerDied","Data":"1c10bfb1b7ec2a16ff2e45a54d644952acfc7da7eeaf6528f0d1db425242027f"} Oct 29 23:48:19.010813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.010734 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc35f7b-ab42-4434-ac06-740df27fdc35" path="/var/lib/kubelet/pods/4bc35f7b-ab42-4434-ac06-740df27fdc35/volumes" Oct 29 23:48:19.084142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.084107 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:19.150686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.150660 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-kube-api-access-cx2gv\" (UniqueName: \"kubernetes.io/projected/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-kube-api-access-cx2gv\") pod \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " Oct 29 23:48:19.150807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.150771 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-xks-ca\") pod \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " Oct 29 23:48:19.151106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.150831 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-openshift-ca\") pod \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\" (UID: \"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73\") " Oct 29 23:48:19.151257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.151231 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-openshift-ca" pod "f5bb76b8-0afa-4e1f-a888-4fb5397d8c73" (UID: "f5bb76b8-0afa-4e1f-a888-4fb5397d8c73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:19.151895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.151736 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.153004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.152982 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-xks-ca" pod "f5bb76b8-0afa-4e1f-a888-4fb5397d8c73" (UID: "f5bb76b8-0afa-4e1f-a888-4fb5397d8c73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:48:19.153346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.153319 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-kube-api-access-cx2gv" pod "f5bb76b8-0afa-4e1f-a888-4fb5397d8c73" (UID: "f5bb76b8-0afa-4e1f-a888-4fb5397d8c73"). InnerVolumeSpecName "kube-api-access-cx2gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:19.170435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.170401 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:48:19.252974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.252940 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istiod-ca-cert\") pod \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " Oct 29 23:48:19.253163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253095 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-certs\") pod \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " Oct 29 23:48:19.253163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253136 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-podinfo\") pod \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " Oct 29 23:48:19.253318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253274 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-data\") pod \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " Oct 29 23:48:19.253383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253338 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-credential-socket\") pod \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " Oct 29 23:48:19.253440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253393 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-token\") pod \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " Oct 29 23:48:19.253440 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253426 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-envoy\") pod \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " Oct 29 23:48:19.253552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253440 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-certs" pod "b9ac2122-30d3-428c-bdcb-906a6ed7e64a" (UID: "b9ac2122-30d3-428c-bdcb-906a6ed7e64a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.253552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253499 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-socket\") pod \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " Oct 29 23:48:19.253552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253504 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istiod-ca-cert" pod "b9ac2122-30d3-428c-bdcb-906a6ed7e64a" (UID: "b9ac2122-30d3-428c-bdcb-906a6ed7e64a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:19.253552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253540 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-data" pod "b9ac2122-30d3-428c-bdcb-906a6ed7e64a" (UID: "b9ac2122-30d3-428c-bdcb-906a6ed7e64a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.253809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253562 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-kube-api-access-js9lj\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-kube-api-access-js9lj\") pod \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\" (UID: \"b9ac2122-30d3-428c-bdcb-906a6ed7e64a\") " Oct 29 23:48:19.253809 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253667 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-credential-socket" pod "b9ac2122-30d3-428c-bdcb-906a6ed7e64a" (UID: "b9ac2122-30d3-428c-bdcb-906a6ed7e64a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.253944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.253810 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-socket" pod "b9ac2122-30d3-428c-bdcb-906a6ed7e64a" (UID: "b9ac2122-30d3-428c-bdcb-906a6ed7e64a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.254434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.254412 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.254434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.254433 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.254650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.254444 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx2gv\" (UniqueName: \"kubernetes.io/projected/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-kube-api-access-cx2gv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.254650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.254453 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.254650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.254463 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.254650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.254472 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.254650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.254481 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.255775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.255738 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-envoy" pod "b9ac2122-30d3-428c-bdcb-906a6ed7e64a" (UID: "b9ac2122-30d3-428c-bdcb-906a6ed7e64a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:19.255955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.255937 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-podinfo" pod "b9ac2122-30d3-428c-bdcb-906a6ed7e64a" (UID: "b9ac2122-30d3-428c-bdcb-906a6ed7e64a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:48:19.256278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.256253 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-kube-api-access-js9lj" pod "b9ac2122-30d3-428c-bdcb-906a6ed7e64a" (UID: "b9ac2122-30d3-428c-bdcb-906a6ed7e64a"). InnerVolumeSpecName "kube-api-access-js9lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:19.256446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.256429 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-token" pod "b9ac2122-30d3-428c-bdcb-906a6ed7e64a" (UID: "b9ac2122-30d3-428c-bdcb-906a6ed7e64a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:19.356058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.356028 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.356058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.356059 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.356246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.356072 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.356246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.356081 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js9lj\" (UniqueName: \"kubernetes.io/projected/b9ac2122-30d3-428c-bdcb-906a6ed7e64a-kube-api-access-js9lj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:19.875814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.875783 5736 generic.go:356] "Generic (PLEG): container finished" podID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerID="b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17" exitCode=0 Oct 29 23:48:19.876004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.875833 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" Oct 29 23:48:19.876004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.875868 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" event={"ID":"b9ac2122-30d3-428c-bdcb-906a6ed7e64a","Type":"ContainerDied","Data":"b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17"} Oct 29 23:48:19.876004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.875939 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6" event={"ID":"b9ac2122-30d3-428c-bdcb-906a6ed7e64a","Type":"ContainerDied","Data":"e2cc6a4dac5f6a26ed42a9da381a4e05fe7a4165e2f6c3f001209186bd1cd426"} Oct 29 23:48:19.876004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.875968 5736 scope.go:122] "RemoveContainer" containerID="b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17" Oct 29 23:48:19.877779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.877756 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" event={"ID":"f5bb76b8-0afa-4e1f-a888-4fb5397d8c73","Type":"ContainerDied","Data":"0d31914f5638f180e8470a615d91bd9a28800c14275b17208f9e50d09813683b"} Oct 29 23:48:19.877922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.877795 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d31914f5638f180e8470a615d91bd9a28800c14275b17208f9e50d09813683b" Oct 29 23:48:19.877922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.877843 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk" Oct 29 23:48:19.887769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.887745 5736 scope.go:122] "RemoveContainer" containerID="b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17" Oct 29 23:48:19.888087 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:48:19.888065 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17\": container with ID starting with b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17 not found: ID does not exist" containerID="b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17" Oct 29 23:48:19.888157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.888095 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17"} err="failed to get container status \"b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17\": rpc error: code = NotFound desc = could not find container \"b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17\": container with ID starting with b455010a30c711d96cf2c1ddd091fc1aa6425155eacb224c5342667f2be20e17 not found: ID does not exist" Oct 29 23:48:19.943552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.943526 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6"] Oct 29 23:48:19.948963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:19.948939 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-fbjc6"] Oct 29 23:48:21.005320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:21.005280 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" path="/var/lib/kubelet/pods/b9ac2122-30d3-428c-bdcb-906a6ed7e64a/volumes" Oct 29 23:48:24.163201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.163173 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:48:24.207338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.207297 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h"] Oct 29 23:48:24.208169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.208144 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f5bb76b8-0afa-4e1f-a888-4fb5397d8c73" containerName="revoke-keys" Oct 29 23:48:24.208302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.208172 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bb76b8-0afa-4e1f-a888-4fb5397d8c73" containerName="revoke-keys" Oct 29 23:48:24.208302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.208212 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" Oct 29 23:48:24.208302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.208220 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" Oct 29 23:48:24.208446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.208386 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bb76b8-0afa-4e1f-a888-4fb5397d8c73" containerName="revoke-keys" Oct 29 23:48:24.208446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.208400 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ac2122-30d3-428c-bdcb-906a6ed7e64a" containerName="istio-proxy" Oct 29 23:48:24.213683 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.213660 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.215385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.215363 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:48:24.234641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.234612 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h"] Oct 29 23:48:24.330502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.330473 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:48:24.330718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.330536 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:48:24.355173 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.355137 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.355338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.355317 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.355400 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.355345 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rwn\" (UniqueName: \"kubernetes.io/projected/7aa8431e-977e-49bd-ada8-a617a8f8122f-kube-api-access-m4rwn\") pod \"maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.477603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.465915 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.477603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.465965 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rwn\" (UniqueName: \"kubernetes.io/projected/7aa8431e-977e-49bd-ada8-a617a8f8122f-kube-api-access-m4rwn\") pod \"maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.477603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.466263 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.477603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.467736 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.488966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.480004 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.513414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.513375 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rwn\" (UniqueName: \"kubernetes.io/projected/7aa8431e-977e-49bd-ada8-a617a8f8122f-kube-api-access-m4rwn\") pod \"maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.532960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.532920 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:24.939224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:24.939179 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h"] Oct 29 23:48:24.942030 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:48:24.941995 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa8431e_977e_49bd_ada8_a617a8f8122f.slice/crio-ca33d3d53ca259c4bbf4f722bd6900a05b48a8a9b8047b3c514fdf61cced4841 WatchSource:0}: Error finding container ca33d3d53ca259c4bbf4f722bd6900a05b48a8a9b8047b3c514fdf61cced4841: Status 404 returned error can't find the container with id ca33d3d53ca259c4bbf4f722bd6900a05b48a8a9b8047b3c514fdf61cced4841 Oct 29 23:48:25.025976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:25.022966 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6"] Oct 29 23:48:25.027554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:25.027529 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l"] Oct 29 23:48:25.027751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:25.027726 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" containerID="cri-o://16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f" gracePeriod=30 Oct 29 23:48:26.040965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.040916 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" event={"ID":"7aa8431e-977e-49bd-ada8-a617a8f8122f","Type":"ContainerStarted","Data":"3d435b3a393decfd05b555e892787264c91beb31e37a364307e0ec218d75a1c7"} Oct 29 23:48:26.041532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.040995 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" event={"ID":"7aa8431e-977e-49bd-ada8-a617a8f8122f","Type":"ContainerStarted","Data":"ca33d3d53ca259c4bbf4f722bd6900a05b48a8a9b8047b3c514fdf61cced4841"} Oct 29 23:48:26.056297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.056263 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79"] Oct 29 23:48:26.062072 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.062037 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899fc1a9-xzm79"] Oct 29 23:48:26.091411 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.091351 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" podStartSLOduration=2.091334224 podStartE2EDuration="2.091334224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:48:26.086098247 +0000 UTC m=+3903.202017854" watchObservedRunningTime="2026-10-29 23:48:26.091334224 +0000 UTC m=+3903.207253860" Oct 29 23:48:26.352695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.352616 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6"] Oct 29 23:48:26.356909 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.356889 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.367814 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.367790 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qng48\" (UniqueName: \"kubernetes.io/projected/d0a8d436-1790-4009-8ca1-63532ae18c0a-kube-api-access-qng48\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.367927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.367852 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.368060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.368043 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.382524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.382496 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6"] Oct 29 23:48:26.469534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.469497 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.469741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.469664 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qng48\" (UniqueName: \"kubernetes.io/projected/d0a8d436-1790-4009-8ca1-63532ae18c0a-kube-api-access-qng48\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.469741 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.469704 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.470384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.470357 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.471930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.471913 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.492642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.492620 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qng48\" (UniqueName: \"kubernetes.io/projected/d0a8d436-1790-4009-8ca1-63532ae18c0a-kube-api-access-qng48\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.672469 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.672380 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:26.828829 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:48:26.828789 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:48:26.984064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:26.984031 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6"] Oct 29 23:48:26.986928 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:48:26.986896 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a8d436_1790_4009_8ca1_63532ae18c0a.slice/crio-72e2b9d6fd0737b005f3660dea371aef08a7d164ede84e78935b20e1efe14fe5 WatchSource:0}: Error finding container 72e2b9d6fd0737b005f3660dea371aef08a7d164ede84e78935b20e1efe14fe5: Status 404 returned error can't find the container with id 72e2b9d6fd0737b005f3660dea371aef08a7d164ede84e78935b20e1efe14fe5 Oct 29 23:48:27.008636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:27.008563 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f" path="/var/lib/kubelet/pods/a9e7ee5f-b2de-4f4a-8a3f-57d761c57c8f/volumes" Oct 29 23:48:27.050676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:27.050643 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" event={"ID":"d0a8d436-1790-4009-8ca1-63532ae18c0a","Type":"ContainerStarted","Data":"72e2b9d6fd0737b005f3660dea371aef08a7d164ede84e78935b20e1efe14fe5"} Oct 29 23:48:27.052325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:27.052299 5736 generic.go:356] "Generic (PLEG): container finished" podID="7aa8431e-977e-49bd-ada8-a617a8f8122f" containerID="3d435b3a393decfd05b555e892787264c91beb31e37a364307e0ec218d75a1c7" exitCode=0 Oct 29 23:48:27.052418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:27.052383 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" event={"ID":"7aa8431e-977e-49bd-ada8-a617a8f8122f","Type":"ContainerDied","Data":"3d435b3a393decfd05b555e892787264c91beb31e37a364307e0ec218d75a1c7"} Oct 29 23:48:28.065475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.065416 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" event={"ID":"d0a8d436-1790-4009-8ca1-63532ae18c0a","Type":"ContainerStarted","Data":"d24cad7f1021d31366007feac07d318bc546abe7ae679e9920a613e8b1e629ac"} Oct 29 23:48:28.110174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.110117 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" podStartSLOduration=2.110099464 podStartE2EDuration="2.110099464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:48:28.106331172 +0000 UTC m=+3905.222250770" watchObservedRunningTime="2026-10-29 23:48:28.110099464 +0000 UTC m=+3905.226019070" Oct 29 23:48:28.206704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.196244 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk"] Oct 29 23:48:28.221976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.217356 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-52f67b-c8fbk"] Oct 29 23:48:28.287964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.287927 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:28.430648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.420195 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa8431e-977e-49bd-ada8-a617a8f8122f-kube-api-access-m4rwn\" (UniqueName: \"kubernetes.io/projected/7aa8431e-977e-49bd-ada8-a617a8f8122f-kube-api-access-m4rwn\") pod \"7aa8431e-977e-49bd-ada8-a617a8f8122f\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " Oct 29 23:48:28.430648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.420257 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-xks-ca\") pod \"7aa8431e-977e-49bd-ada8-a617a8f8122f\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " Oct 29 23:48:28.430648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.420358 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-openshift-ca\") pod \"7aa8431e-977e-49bd-ada8-a617a8f8122f\" (UID: \"7aa8431e-977e-49bd-ada8-a617a8f8122f\") " Oct 29 23:48:28.430648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.428753 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-openshift-ca" pod "7aa8431e-977e-49bd-ada8-a617a8f8122f" (UID: "7aa8431e-977e-49bd-ada8-a617a8f8122f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:28.438727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.438448 5736 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:48:28.438727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.438500 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:48:28.441451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.441415 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-xks-ca" pod "7aa8431e-977e-49bd-ada8-a617a8f8122f" (UID: "7aa8431e-977e-49bd-ada8-a617a8f8122f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:48:28.442071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.442045 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa8431e-977e-49bd-ada8-a617a8f8122f-kube-api-access-m4rwn" pod "7aa8431e-977e-49bd-ada8-a617a8f8122f" (UID: "7aa8431e-977e-49bd-ada8-a617a8f8122f"). InnerVolumeSpecName "kube-api-access-m4rwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:28.522374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.522344 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:28.522374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.522371 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4rwn\" (UniqueName: \"kubernetes.io/projected/7aa8431e-977e-49bd-ada8-a617a8f8122f-kube-api-access-m4rwn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:28.522557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:28.522382 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa8431e-977e-49bd-ada8-a617a8f8122f-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:29.008635 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:29.008589 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5bb76b8-0afa-4e1f-a888-4fb5397d8c73" path="/var/lib/kubelet/pods/f5bb76b8-0afa-4e1f-a888-4fb5397d8c73/volumes" Oct 29 23:48:29.074501 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:29.074468 5736 generic.go:356] "Generic (PLEG): container finished" podID="d0a8d436-1790-4009-8ca1-63532ae18c0a" containerID="d24cad7f1021d31366007feac07d318bc546abe7ae679e9920a613e8b1e629ac" exitCode=0 Oct 29 23:48:29.074966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:29.074552 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" event={"ID":"d0a8d436-1790-4009-8ca1-63532ae18c0a","Type":"ContainerDied","Data":"d24cad7f1021d31366007feac07d318bc546abe7ae679e9920a613e8b1e629ac"} Oct 29 23:48:29.076086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:29.076061 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" Oct 29 23:48:29.076212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:29.076089 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h" event={"ID":"7aa8431e-977e-49bd-ada8-a617a8f8122f","Type":"ContainerDied","Data":"ca33d3d53ca259c4bbf4f722bd6900a05b48a8a9b8047b3c514fdf61cced4841"} Oct 29 23:48:29.076212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:29.076132 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca33d3d53ca259c4bbf4f722bd6900a05b48a8a9b8047b3c514fdf61cced4841" Oct 29 23:48:30.023919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.023881 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" podUID="7fae0669-43fb-4822-bc69-9556cb2f0426" containerName="payload-processing" containerID="cri-o://b09f70f23eca73fc10060b4b0e5adc4151b5fbee424cd9594c7f6c5b6164205e" gracePeriod=25 Oct 29 23:48:30.313960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.313931 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:30.332894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.332828 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" podUID="7fae0669-43fb-4822-bc69-9556cb2f0426" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:48:30.368090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.368027 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-openshift-ca\") pod \"d0a8d436-1790-4009-8ca1-63532ae18c0a\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " Oct 29 23:48:30.368289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.368269 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a8d436-1790-4009-8ca1-63532ae18c0a-kube-api-access-qng48\" (UniqueName: \"kubernetes.io/projected/d0a8d436-1790-4009-8ca1-63532ae18c0a-kube-api-access-qng48\") pod \"d0a8d436-1790-4009-8ca1-63532ae18c0a\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " Oct 29 23:48:30.368427 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.368386 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-xks-ca\") pod \"d0a8d436-1790-4009-8ca1-63532ae18c0a\" (UID: \"d0a8d436-1790-4009-8ca1-63532ae18c0a\") " Oct 29 23:48:30.369852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.369817 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-openshift-ca" pod "d0a8d436-1790-4009-8ca1-63532ae18c0a" (UID: "d0a8d436-1790-4009-8ca1-63532ae18c0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:30.372641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.372612 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-xks-ca" pod "d0a8d436-1790-4009-8ca1-63532ae18c0a" (UID: "d0a8d436-1790-4009-8ca1-63532ae18c0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:48:30.387717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.387685 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a8d436-1790-4009-8ca1-63532ae18c0a-kube-api-access-qng48" pod "d0a8d436-1790-4009-8ca1-63532ae18c0a" (UID: "d0a8d436-1790-4009-8ca1-63532ae18c0a"). InnerVolumeSpecName "kube-api-access-qng48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:30.437946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.437908 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 23:48:30.470054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470025 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f61bd156-2fc9-427e-9ae6-224a79565512-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f61bd156-2fc9-427e-9ae6-224a79565512-istio-podinfo\") pod \"f61bd156-2fc9-427e-9ae6-224a79565512\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " Oct 29 23:48:30.470186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470084 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-istio-token\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-istio-token\") pod \"f61bd156-2fc9-427e-9ae6-224a79565512\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " Oct 29 23:48:30.470273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470257 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f61bd156-2fc9-427e-9ae6-224a79565512-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f61bd156-2fc9-427e-9ae6-224a79565512-istiod-ca-cert\") pod \"f61bd156-2fc9-427e-9ae6-224a79565512\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " Oct 29 23:48:30.470332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470316 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-data\") pod \"f61bd156-2fc9-427e-9ae6-224a79565512\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " Oct 29 23:48:30.470449 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470428 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-envoy\") pod \"f61bd156-2fc9-427e-9ae6-224a79565512\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " Oct 29 23:48:30.470542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470473 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-kube-api-access-w9c7z\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-kube-api-access-w9c7z\") pod \"f61bd156-2fc9-427e-9ae6-224a79565512\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " Oct 29 23:48:30.470542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470531 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-socket\") pod \"f61bd156-2fc9-427e-9ae6-224a79565512\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " Oct 29 23:48:30.470705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470560 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-certs\") pod \"f61bd156-2fc9-427e-9ae6-224a79565512\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " Oct 29 23:48:30.470705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470643 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-credential-socket\") pod \"f61bd156-2fc9-427e-9ae6-224a79565512\" (UID: \"f61bd156-2fc9-427e-9ae6-224a79565512\") " Oct 29 23:48:30.470705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470648 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-data" pod "f61bd156-2fc9-427e-9ae6-224a79565512" (UID: "f61bd156-2fc9-427e-9ae6-224a79565512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:30.470705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470682 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f61bd156-2fc9-427e-9ae6-224a79565512-istiod-ca-cert" pod "f61bd156-2fc9-427e-9ae6-224a79565512" (UID: "f61bd156-2fc9-427e-9ae6-224a79565512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:30.470937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.470887 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-socket" pod "f61bd156-2fc9-427e-9ae6-224a79565512" (UID: "f61bd156-2fc9-427e-9ae6-224a79565512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:30.471150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471117 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-credential-socket" pod "f61bd156-2fc9-427e-9ae6-224a79565512" (UID: "f61bd156-2fc9-427e-9ae6-224a79565512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:30.471267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471151 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-certs" pod "f61bd156-2fc9-427e-9ae6-224a79565512" (UID: "f61bd156-2fc9-427e-9ae6-224a79565512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:30.471633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471607 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.471633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471634 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.471828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471645 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.471828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471655 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qng48\" (UniqueName: \"kubernetes.io/projected/d0a8d436-1790-4009-8ca1-63532ae18c0a-kube-api-access-qng48\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.471828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471664 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.471828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471673 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a8d436-1790-4009-8ca1-63532ae18c0a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.471828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471681 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f61bd156-2fc9-427e-9ae6-224a79565512-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.471828 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.471689 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.472850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.472820 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f61bd156-2fc9-427e-9ae6-224a79565512-istio-podinfo" pod "f61bd156-2fc9-427e-9ae6-224a79565512" (UID: "f61bd156-2fc9-427e-9ae6-224a79565512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:48:30.472953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.472865 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-istio-token" pod "f61bd156-2fc9-427e-9ae6-224a79565512" (UID: "f61bd156-2fc9-427e-9ae6-224a79565512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:30.472953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.472941 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-envoy" pod "f61bd156-2fc9-427e-9ae6-224a79565512" (UID: "f61bd156-2fc9-427e-9ae6-224a79565512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:30.473260 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.473238 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-kube-api-access-w9c7z" pod "f61bd156-2fc9-427e-9ae6-224a79565512" (UID: "f61bd156-2fc9-427e-9ae6-224a79565512"). InnerVolumeSpecName "kube-api-access-w9c7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:30.573012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.572930 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f61bd156-2fc9-427e-9ae6-224a79565512-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.573012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.572967 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.573012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.572978 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f61bd156-2fc9-427e-9ae6-224a79565512-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:30.573012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:30.572988 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9c7z\" (UniqueName: \"kubernetes.io/projected/f61bd156-2fc9-427e-9ae6-224a79565512-kube-api-access-w9c7z\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:31.094360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.094329 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" Oct 29 23:48:31.094487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.094359 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6" event={"ID":"d0a8d436-1790-4009-8ca1-63532ae18c0a","Type":"ContainerDied","Data":"72e2b9d6fd0737b005f3660dea371aef08a7d164ede84e78935b20e1efe14fe5"} Oct 29 23:48:31.094487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.094410 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72e2b9d6fd0737b005f3660dea371aef08a7d164ede84e78935b20e1efe14fe5" Oct 29 23:48:31.095929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.095900 5736 generic.go:356] "Generic (PLEG): container finished" podID="f61bd156-2fc9-427e-9ae6-224a79565512" containerID="16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f" exitCode=0 Oct 29 23:48:31.096059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.095936 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" Oct 29 23:48:31.096059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.095988 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" event={"ID":"f61bd156-2fc9-427e-9ae6-224a79565512","Type":"ContainerDied","Data":"16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f"} Oct 29 23:48:31.096059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.096054 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l" event={"ID":"f61bd156-2fc9-427e-9ae6-224a79565512","Type":"ContainerDied","Data":"843a5a143cd05e34ef6b386295a8846744ecafd33dae6d62111231f9d87f9f1b"} Oct 29 23:48:31.096232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.096085 5736 scope.go:122] "RemoveContainer" containerID="16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f" Oct 29 23:48:31.107412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.107391 5736 scope.go:122] "RemoveContainer" containerID="16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f" Oct 29 23:48:31.107722 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:48:31.107696 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f\": container with ID starting with 16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f not found: ID does not exist" containerID="16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f" Oct 29 23:48:31.107791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.107729 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f"} err="failed to get container status \"16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f\": rpc error: code = NotFound desc = could not find container \"16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f\": container with ID starting with 16f1d8d4f1cd0c3d026878da45d9798537100eaf846c537764ecbf92aeb06b7f not found: ID does not exist" Oct 29 23:48:31.114421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.114397 5736 scope.go:122] "RemoveContainer" containerID="ee639e4dfca1fc57ee8e9456df87c0d84a41ab1b331dd4dc08390e6d0159af67" Oct 29 23:48:31.152500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.152472 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l"] Oct 29 23:48:31.163241 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:31.163213 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-4mv7l"] Oct 29 23:48:33.006044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:33.006010 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" path="/var/lib/kubelet/pods/f61bd156-2fc9-427e-9ae6-224a79565512/volumes" Oct 29 23:48:40.332965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:40.332925 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" podUID="7fae0669-43fb-4822-bc69-9556cb2f0426" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:48:40.441705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:40.441668 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6"] Oct 29 23:48:40.445137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:40.445110 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4ad17e-ggzv6"] Oct 29 23:48:41.006881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:41.006844 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a8d436-1790-4009-8ca1-63532ae18c0a" path="/var/lib/kubelet/pods/d0a8d436-1790-4009-8ca1-63532ae18c0a/volumes" Oct 29 23:48:45.840150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:45.840111 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x"] Oct 29 23:48:45.840659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:45.840383 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" containerID="cri-o://6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8" gracePeriod=30 Oct 29 23:48:46.285092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:46.285045 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:48:46.285268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:46.285119 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:48:48.511697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.511660 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c"] Oct 29 23:48:48.512334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.512318 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" Oct 29 23:48:48.512334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.512333 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" Oct 29 23:48:48.512475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.512359 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d0a8d436-1790-4009-8ca1-63532ae18c0a" containerName="revoke-keys" Oct 29 23:48:48.512475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.512365 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a8d436-1790-4009-8ca1-63532ae18c0a" containerName="revoke-keys" Oct 29 23:48:48.512475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.512375 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7aa8431e-977e-49bd-ada8-a617a8f8122f" containerName="revoke-keys" Oct 29 23:48:48.512475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.512381 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa8431e-977e-49bd-ada8-a617a8f8122f" containerName="revoke-keys" Oct 29 23:48:48.512669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.512502 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61bd156-2fc9-427e-9ae6-224a79565512" containerName="istio-proxy" Oct 29 23:48:48.512669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.512514 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a8d436-1790-4009-8ca1-63532ae18c0a" containerName="revoke-keys" Oct 29 23:48:48.512669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.512520 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa8431e-977e-49bd-ada8-a617a8f8122f" containerName="revoke-keys" Oct 29 23:48:48.518003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.517978 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:48.565507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.565478 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:48:48.565712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.565474 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:48:48.570935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.570904 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c"] Oct 29 23:48:48.709504 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.709469 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:48.709729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.709532 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:48.709729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.709694 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9q6\" (UniqueName: \"kubernetes.io/projected/9db88598-c68d-423e-ac5b-4cb29ea4d52e-kube-api-access-jq9q6\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:48.811174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.811082 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:48.811174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.811148 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:48.811407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.811197 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9q6\" (UniqueName: \"kubernetes.io/projected/9db88598-c68d-423e-ac5b-4cb29ea4d52e-kube-api-access-jq9q6\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:48.811856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.811831 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:48.813671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.813643 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:48.838558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:48.838530 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9q6\" (UniqueName: \"kubernetes.io/projected/9db88598-c68d-423e-ac5b-4cb29ea4d52e-kube-api-access-jq9q6\") pod \"maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:49.133022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:49.132949 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:49.445959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:49.445934 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c"] Oct 29 23:48:49.448573 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:48:49.448544 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db88598_c68d_423e_ac5b_4cb29ea4d52e.slice/crio-c53f6ae2c6c69b9ce0923acb88695da31fa5fae65fb63202321b92c30e8404f1 WatchSource:0}: Error finding container c53f6ae2c6c69b9ce0923acb88695da31fa5fae65fb63202321b92c30e8404f1: Status 404 returned error can't find the container with id c53f6ae2c6c69b9ce0923acb88695da31fa5fae65fb63202321b92c30e8404f1 Oct 29 23:48:50.265946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.265912 5736 generic.go:356] "Generic (PLEG): container finished" podID="7fae0669-43fb-4822-bc69-9556cb2f0426" containerID="b09f70f23eca73fc10060b4b0e5adc4151b5fbee424cd9594c7f6c5b6164205e" exitCode=0 Oct 29 23:48:50.266432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.265986 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" event={"ID":"7fae0669-43fb-4822-bc69-9556cb2f0426","Type":"ContainerDied","Data":"b09f70f23eca73fc10060b4b0e5adc4151b5fbee424cd9594c7f6c5b6164205e"} Oct 29 23:48:50.267910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.267879 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" event={"ID":"9db88598-c68d-423e-ac5b-4cb29ea4d52e","Type":"ContainerStarted","Data":"831d96c0b5433fb6a6292a7490960f90c53389a114ba6d390adbfa04bac06f7f"} Oct 29 23:48:50.268057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.267921 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" event={"ID":"9db88598-c68d-423e-ac5b-4cb29ea4d52e","Type":"ContainerStarted","Data":"c53f6ae2c6c69b9ce0923acb88695da31fa5fae65fb63202321b92c30e8404f1"} Oct 29 23:48:50.310932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.310909 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:48:50.316014 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.315955 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" podStartSLOduration=2.315939868 podStartE2EDuration="2.315939868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:48:50.314656465 +0000 UTC m=+3927.430576070" watchObservedRunningTime="2026-10-29 23:48:50.315939868 +0000 UTC m=+3927.431859475" Oct 29 23:48:50.441924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.440160 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fae0669-43fb-4822-bc69-9556cb2f0426-kube-api-access-rgm2t\" (UniqueName: \"kubernetes.io/projected/7fae0669-43fb-4822-bc69-9556cb2f0426-kube-api-access-rgm2t\") pod \"7fae0669-43fb-4822-bc69-9556cb2f0426\" (UID: \"7fae0669-43fb-4822-bc69-9556cb2f0426\") " Oct 29 23:48:50.441924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.440219 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fae0669-43fb-4822-bc69-9556cb2f0426-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fae0669-43fb-4822-bc69-9556cb2f0426-plugins-config-volume\") pod \"7fae0669-43fb-4822-bc69-9556cb2f0426\" (UID: \"7fae0669-43fb-4822-bc69-9556cb2f0426\") " Oct 29 23:48:50.441924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.441397 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fae0669-43fb-4822-bc69-9556cb2f0426-plugins-config-volume" pod "7fae0669-43fb-4822-bc69-9556cb2f0426" (UID: "7fae0669-43fb-4822-bc69-9556cb2f0426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:50.441924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.441901 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fae0669-43fb-4822-bc69-9556cb2f0426-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:50.443306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.443260 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fae0669-43fb-4822-bc69-9556cb2f0426-kube-api-access-rgm2t" pod "7fae0669-43fb-4822-bc69-9556cb2f0426" (UID: "7fae0669-43fb-4822-bc69-9556cb2f0426"). InnerVolumeSpecName "kube-api-access-rgm2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:50.543697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:50.543628 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgm2t\" (UniqueName: \"kubernetes.io/projected/7fae0669-43fb-4822-bc69-9556cb2f0426-kube-api-access-rgm2t\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.131659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.131632 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:48:51.151382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151355 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-podinfo\") pod \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " Oct 29 23:48:51.151507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151414 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-socket\") pod \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " Oct 29 23:48:51.151547 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151526 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-kube-api-access-xn5z8\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-kube-api-access-xn5z8\") pod \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " Oct 29 23:48:51.151600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151562 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istiod-ca-cert\") pod \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " Oct 29 23:48:51.151649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151609 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-envoy\") pod \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " Oct 29 23:48:51.151724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151698 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-data\") pod \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " Oct 29 23:48:51.151790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151754 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-socket" pod "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" (UID: "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:51.151855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151787 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-certs\") pod \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " Oct 29 23:48:51.151925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151915 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-token\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-token\") pod \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " Oct 29 23:48:51.151985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151959 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-credential-socket\") pod \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\" (UID: \"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4\") " Oct 29 23:48:51.152051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.151976 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istiod-ca-cert" pod "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" (UID: "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:51.152051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.152003 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-data" pod "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" (UID: "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:51.152177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.152105 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-certs" pod "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" (UID: "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:51.152294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.152272 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-credential-socket" pod "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" (UID: "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:51.153071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.153043 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.153201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.153074 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.153201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.153089 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.153201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.153102 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.153201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.153116 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.153991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.153960 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-podinfo" pod "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" (UID: "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:48:51.154160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.154143 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-envoy" pod "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" (UID: "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:48:51.154215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.154176 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-kube-api-access-xn5z8" pod "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" (UID: "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4"). InnerVolumeSpecName "kube-api-access-xn5z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:51.154336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.154318 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-token" pod "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" (UID: "41cdd1d9-d919-4a0f-95f5-59e338d9d8e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:51.254766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.254731 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.254766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.254767 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.254955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.254783 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn5z8\" (UniqueName: \"kubernetes.io/projected/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-kube-api-access-xn5z8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.254955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.254799 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:51.278391 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.278355 5736 generic.go:356] "Generic (PLEG): container finished" podID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerID="6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8" exitCode=0 Oct 29 23:48:51.278818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.278414 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" Oct 29 23:48:51.278818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.278433 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" event={"ID":"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4","Type":"ContainerDied","Data":"6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8"} Oct 29 23:48:51.278818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.278484 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x" event={"ID":"41cdd1d9-d919-4a0f-95f5-59e338d9d8e4","Type":"ContainerDied","Data":"8aaf21a2ec7c2ca070c8adb105b3a255d280da9f2ce81124e137f9baa9e68069"} Oct 29 23:48:51.278818 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.278498 5736 scope.go:122] "RemoveContainer" containerID="6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8" Oct 29 23:48:51.280229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.280209 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" Oct 29 23:48:51.280346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.280241 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6" event={"ID":"7fae0669-43fb-4822-bc69-9556cb2f0426","Type":"ContainerDied","Data":"7c3058e75bcad3ab19d447b514c28b9a40a10c4ae17c9a4e398b16b30bb70d2e"} Oct 29 23:48:51.282299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.282275 5736 generic.go:356] "Generic (PLEG): container finished" podID="9db88598-c68d-423e-ac5b-4cb29ea4d52e" containerID="831d96c0b5433fb6a6292a7490960f90c53389a114ba6d390adbfa04bac06f7f" exitCode=0 Oct 29 23:48:51.282409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.282338 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" event={"ID":"9db88598-c68d-423e-ac5b-4cb29ea4d52e","Type":"ContainerDied","Data":"831d96c0b5433fb6a6292a7490960f90c53389a114ba6d390adbfa04bac06f7f"} Oct 29 23:48:51.290954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.290935 5736 scope.go:122] "RemoveContainer" containerID="5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955" Oct 29 23:48:51.301425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.301406 5736 scope.go:122] "RemoveContainer" containerID="6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8" Oct 29 23:48:51.301725 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:48:51.301700 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8\": container with ID starting with 6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8 not found: ID does not exist" containerID="6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8" Oct 29 23:48:51.301782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.301734 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8"} err="failed to get container status \"6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8\": rpc error: code = NotFound desc = could not find container \"6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8\": container with ID starting with 6bd1b58d6b2b1489978a1f5c6e411145752ddc92e028c1a19c7defbef80405f8 not found: ID does not exist" Oct 29 23:48:51.301782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.301750 5736 scope.go:122] "RemoveContainer" containerID="5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955" Oct 29 23:48:51.302035 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:48:51.302016 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955\": container with ID starting with 5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955 not found: ID does not exist" containerID="5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955" Oct 29 23:48:51.302099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.302040 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955"} err="failed to get container status \"5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955\": rpc error: code = NotFound desc = could not find container \"5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955\": container with ID starting with 5d3bb1f133f7a6879bab9d97cce4a29eb6d2e156ccb95d749f3d5932d38c8955 not found: ID does not exist" Oct 29 23:48:51.302099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.302052 5736 scope.go:122] "RemoveContainer" containerID="b09f70f23eca73fc10060b4b0e5adc4151b5fbee424cd9594c7f6c5b6164205e" Oct 29 23:48:51.337422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.337397 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6"] Oct 29 23:48:51.356500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.356435 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-52f67b-699484646b-ckjq6"] Oct 29 23:48:51.406262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.397669 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp"] Oct 29 23:48:51.439321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.439294 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww"] Oct 29 23:48:51.459764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.459734 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv"] Oct 29 23:48:51.468800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.468769 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x"] Oct 29 23:48:51.486412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:51.486382 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-52f67b-openshift-default-d46949685-qt97x"] Oct 29 23:48:52.509746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.509715 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:52.575756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.575675 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-openshift-ca\") pod \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " Oct 29 23:48:52.575975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.575795 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db88598-c68d-423e-ac5b-4cb29ea4d52e-kube-api-access-jq9q6\" (UniqueName: \"kubernetes.io/projected/9db88598-c68d-423e-ac5b-4cb29ea4d52e-kube-api-access-jq9q6\") pod \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " Oct 29 23:48:52.576056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.576016 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-xks-ca\") pod \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\" (UID: \"9db88598-c68d-423e-ac5b-4cb29ea4d52e\") " Oct 29 23:48:52.576121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.576073 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-openshift-ca" pod "9db88598-c68d-423e-ac5b-4cb29ea4d52e" (UID: "9db88598-c68d-423e-ac5b-4cb29ea4d52e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:52.576697 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.576667 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:52.578474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.578442 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db88598-c68d-423e-ac5b-4cb29ea4d52e-kube-api-access-jq9q6" pod "9db88598-c68d-423e-ac5b-4cb29ea4d52e" (UID: "9db88598-c68d-423e-ac5b-4cb29ea4d52e"). InnerVolumeSpecName "kube-api-access-jq9q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:52.578561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.578479 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-xks-ca" pod "9db88598-c68d-423e-ac5b-4cb29ea4d52e" (UID: "9db88598-c68d-423e-ac5b-4cb29ea4d52e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:48:52.677958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.677923 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq9q6\" (UniqueName: \"kubernetes.io/projected/9db88598-c68d-423e-ac5b-4cb29ea4d52e-kube-api-access-jq9q6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:52.677958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:52.677953 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db88598-c68d-423e-ac5b-4cb29ea4d52e-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:53.004743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:53.004709 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" path="/var/lib/kubelet/pods/41cdd1d9-d919-4a0f-95f5-59e338d9d8e4/volumes" Oct 29 23:48:53.005166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:53.005152 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fae0669-43fb-4822-bc69-9556cb2f0426" path="/var/lib/kubelet/pods/7fae0669-43fb-4822-bc69-9556cb2f0426/volumes" Oct 29 23:48:53.201018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:53.200971 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh"] Oct 29 23:48:53.213170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:53.213139 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz"] Oct 29 23:48:53.228684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:53.228657 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw"] Oct 29 23:48:53.302372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:53.302297 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" Oct 29 23:48:53.302526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:53.302293 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c" event={"ID":"9db88598-c68d-423e-ac5b-4cb29ea4d52e","Type":"ContainerDied","Data":"c53f6ae2c6c69b9ce0923acb88695da31fa5fae65fb63202321b92c30e8404f1"} Oct 29 23:48:53.302526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:53.302438 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c53f6ae2c6c69b9ce0923acb88695da31fa5fae65fb63202321b92c30e8404f1" Oct 29 23:48:56.644159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.644123 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk"] Oct 29 23:48:56.644996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.644971 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7fae0669-43fb-4822-bc69-9556cb2f0426" containerName="payload-processing" Oct 29 23:48:56.645120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.644996 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fae0669-43fb-4822-bc69-9556cb2f0426" containerName="payload-processing" Oct 29 23:48:56.645120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645026 5736 cpu_manager.go:414] "containerMap: removing container" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" Oct 29 23:48:56.645120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645035 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" Oct 29 23:48:56.645120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645069 5736 cpu_manager.go:414] "containerMap: removing container" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" Oct 29 23:48:56.645120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645076 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" Oct 29 23:48:56.645120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645103 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9db88598-c68d-423e-ac5b-4cb29ea4d52e" containerName="revoke-keys" Oct 29 23:48:56.645120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645110 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db88598-c68d-423e-ac5b-4cb29ea4d52e" containerName="revoke-keys" Oct 29 23:48:56.645493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645308 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fae0669-43fb-4822-bc69-9556cb2f0426" containerName="payload-processing" Oct 29 23:48:56.645493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645330 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" Oct 29 23:48:56.645493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645344 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cdd1d9-d919-4a0f-95f5-59e338d9d8e4" containerName="istio-proxy" Oct 29 23:48:56.645493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.645358 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db88598-c68d-423e-ac5b-4cb29ea4d52e" containerName="revoke-keys" Oct 29 23:48:56.650745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.650712 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.676401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.676374 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:48:56.685379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.685359 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:48:56.693066 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.693040 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk"] Oct 29 23:48:56.734265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.734235 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7j6n\" (UniqueName: \"kubernetes.io/projected/af2d7af5-e65f-47c1-9323-9851b1721066-kube-api-access-g7j6n\") pod \"maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.734410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.734378 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.734459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.734423 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.825266 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:48:56.825200 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:48:56.836157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.836123 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.836291 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.836172 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.836385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.836339 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7j6n\" (UniqueName: \"kubernetes.io/projected/af2d7af5-e65f-47c1-9323-9851b1721066-kube-api-access-g7j6n\") pod \"maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.836869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.836848 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.838615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.838555 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.862373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.862344 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7j6n\" (UniqueName: \"kubernetes.io/projected/af2d7af5-e65f-47c1-9323-9851b1721066-kube-api-access-g7j6n\") pod \"maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:56.966551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:56.966524 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:57.340775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:57.340738 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk"] Oct 29 23:48:57.341722 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:48:57.341680 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2d7af5_e65f_47c1_9323_9851b1721066.slice/crio-1d6bd3e7947baa8b76ee91d75760310894531b9ba0df1c3f06868ac8efa40701 WatchSource:0}: Error finding container 1d6bd3e7947baa8b76ee91d75760310894531b9ba0df1c3f06868ac8efa40701: Status 404 returned error can't find the container with id 1d6bd3e7947baa8b76ee91d75760310894531b9ba0df1c3f06868ac8efa40701 Oct 29 23:48:58.352329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:58.352294 5736 generic.go:356] "Generic (PLEG): container finished" podID="af2d7af5-e65f-47c1-9323-9851b1721066" containerID="d489217f5a7a4aa63c5f615f0c2d93c2ec89cdc977ce31029b95631379f8cfb6" exitCode=0 Oct 29 23:48:58.352800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:58.352397 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" event={"ID":"af2d7af5-e65f-47c1-9323-9851b1721066","Type":"ContainerDied","Data":"d489217f5a7a4aa63c5f615f0c2d93c2ec89cdc977ce31029b95631379f8cfb6"} Oct 29 23:48:58.352800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:58.352468 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" event={"ID":"af2d7af5-e65f-47c1-9323-9851b1721066","Type":"ContainerStarted","Data":"1d6bd3e7947baa8b76ee91d75760310894531b9ba0df1c3f06868ac8efa40701"} Oct 29 23:48:59.516567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.516539 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:48:59.660614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.628673 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-xks-ca\") pod \"af2d7af5-e65f-47c1-9323-9851b1721066\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " Oct 29 23:48:59.660614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.628841 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2d7af5-e65f-47c1-9323-9851b1721066-kube-api-access-g7j6n\" (UniqueName: \"kubernetes.io/projected/af2d7af5-e65f-47c1-9323-9851b1721066-kube-api-access-g7j6n\") pod \"af2d7af5-e65f-47c1-9323-9851b1721066\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " Oct 29 23:48:59.660614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.629113 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-openshift-ca\") pod \"af2d7af5-e65f-47c1-9323-9851b1721066\" (UID: \"af2d7af5-e65f-47c1-9323-9851b1721066\") " Oct 29 23:48:59.660614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.654817 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-openshift-ca" pod "af2d7af5-e65f-47c1-9323-9851b1721066" (UID: "af2d7af5-e65f-47c1-9323-9851b1721066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:48:59.700876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.700820 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2d7af5-e65f-47c1-9323-9851b1721066-kube-api-access-g7j6n" pod "af2d7af5-e65f-47c1-9323-9851b1721066" (UID: "af2d7af5-e65f-47c1-9323-9851b1721066"). InnerVolumeSpecName "kube-api-access-g7j6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:48:59.706795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.706757 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-xks-ca" pod "af2d7af5-e65f-47c1-9323-9851b1721066" (UID: "af2d7af5-e65f-47c1-9323-9851b1721066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:48:59.734600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.734396 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:59.734600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.734439 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7j6n\" (UniqueName: \"kubernetes.io/projected/af2d7af5-e65f-47c1-9323-9851b1721066-kube-api-access-g7j6n\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:48:59.734600 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:48:59.734457 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2d7af5-e65f-47c1-9323-9851b1721066-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:00.369803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:00.369770 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" Oct 29 23:49:00.369966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:00.369807 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk" event={"ID":"af2d7af5-e65f-47c1-9323-9851b1721066","Type":"ContainerDied","Data":"1d6bd3e7947baa8b76ee91d75760310894531b9ba0df1c3f06868ac8efa40701"} Oct 29 23:49:00.369966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:00.369854 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d6bd3e7947baa8b76ee91d75760310894531b9ba0df1c3f06868ac8efa40701" Oct 29 23:49:06.425964 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.425908 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" containerID="cri-o://45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b" gracePeriod=45 Oct 29 23:49:06.463125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.463077 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" containerID="cri-o://60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709" gracePeriod=45 Oct 29 23:49:06.482357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.482312 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" containerID="cri-o://552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e" gracePeriod=45 Oct 29 23:49:06.664774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.663960 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused" Oct 29 23:49:06.838603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.838564 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:49:06.846554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.846527 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 23:49:06.855911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.855882 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:49:06.942201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942163 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960f35db-200b-4e69-ae76-34f118f1af31-kube-api-access-vwknv\" (UniqueName: \"kubernetes.io/projected/960f35db-200b-4e69-ae76-34f118f1af31-kube-api-access-vwknv\") pod \"960f35db-200b-4e69-ae76-34f118f1af31\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " Oct 29 23:49:06.942441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942223 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-model-cache\") pod \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " Oct 29 23:49:06.942441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942260 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-dshm\") pod \"fcd831ca-e923-49fb-9993-fb6954bf068b\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " Oct 29 23:49:06.942441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942355 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/960f35db-200b-4e69-ae76-34f118f1af31-tls-certs\" (UniqueName: \"kubernetes.io/secret/960f35db-200b-4e69-ae76-34f118f1af31-tls-certs\") pod \"960f35db-200b-4e69-ae76-34f118f1af31\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " Oct 29 23:49:06.942441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942408 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcd831ca-e923-49fb-9993-fb6954bf068b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd831ca-e923-49fb-9993-fb6954bf068b-tls-certs\") pod \"fcd831ca-e923-49fb-9993-fb6954bf068b\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " Oct 29 23:49:06.942725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942445 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-home\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-home\") pod \"960f35db-200b-4e69-ae76-34f118f1af31\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " Oct 29 23:49:06.942725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942473 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-model-cache\") pod \"960f35db-200b-4e69-ae76-34f118f1af31\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " Oct 29 23:49:06.942725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942553 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-tmp-dir\") pod \"960f35db-200b-4e69-ae76-34f118f1af31\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " Oct 29 23:49:06.942725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942634 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tls-certs\") pod \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " Oct 29 23:49:06.942725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942666 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-tmp-dir\") pod \"fcd831ca-e923-49fb-9993-fb6954bf068b\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " Oct 29 23:49:06.942725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942696 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd831ca-e923-49fb-9993-fb6954bf068b-kube-api-access-xxd9v\" (UniqueName: \"kubernetes.io/projected/fcd831ca-e923-49fb-9993-fb6954bf068b-kube-api-access-xxd9v\") pod \"fcd831ca-e923-49fb-9993-fb6954bf068b\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942748 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-home\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-home\") pod \"fcd831ca-e923-49fb-9993-fb6954bf068b\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942779 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-home\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-home\") pod \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942810 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-dshm\") pod \"960f35db-200b-4e69-ae76-34f118f1af31\" (UID: \"960f35db-200b-4e69-ae76-34f118f1af31\") " Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942858 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-home" pod "960f35db-200b-4e69-ae76-34f118f1af31" (UID: "960f35db-200b-4e69-ae76-34f118f1af31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.942960 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-model-cache" pod "0bcce0f2-4dd8-4022-933f-e826e3b07b11" (UID: "0bcce0f2-4dd8-4022-933f-e826e3b07b11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.943419 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-model-cache" pod "960f35db-200b-4e69-ae76-34f118f1af31" (UID: "960f35db-200b-4e69-ae76-34f118f1af31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.943674 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-home" pod "fcd831ca-e923-49fb-9993-fb6954bf068b" (UID: "fcd831ca-e923-49fb-9993-fb6954bf068b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.944113 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-home" pod "0bcce0f2-4dd8-4022-933f-e826e3b07b11" (UID: "0bcce0f2-4dd8-4022-933f-e826e3b07b11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.944404 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-tmp-dir" pod "960f35db-200b-4e69-ae76-34f118f1af31" (UID: "960f35db-200b-4e69-ae76-34f118f1af31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.944509 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-model-cache\") pod \"fcd831ca-e923-49fb-9993-fb6954bf068b\" (UID: \"fcd831ca-e923-49fb-9993-fb6954bf068b\") " Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.944570 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tmp-dir\") pod \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.944691 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-dshm\") pod \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.944728 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcce0f2-4dd8-4022-933f-e826e3b07b11-kube-api-access-gsrpt\" (UniqueName: \"kubernetes.io/projected/0bcce0f2-4dd8-4022-933f-e826e3b07b11-kube-api-access-gsrpt\") pod \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\" (UID: \"0bcce0f2-4dd8-4022-933f-e826e3b07b11\") " Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.945710 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.945734 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.945748 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.945760 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.945772 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:06.946186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.945784 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:06.947396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.946633 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tmp-dir" pod "0bcce0f2-4dd8-4022-933f-e826e3b07b11" (UID: "0bcce0f2-4dd8-4022-933f-e826e3b07b11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.947396 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.946819 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-model-cache" pod "fcd831ca-e923-49fb-9993-fb6954bf068b" (UID: "fcd831ca-e923-49fb-9993-fb6954bf068b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.947536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.947507 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-tmp-dir" pod "fcd831ca-e923-49fb-9993-fb6954bf068b" (UID: "fcd831ca-e923-49fb-9993-fb6954bf068b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.948006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.947969 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tls-certs" pod "0bcce0f2-4dd8-4022-933f-e826e3b07b11" (UID: "0bcce0f2-4dd8-4022-933f-e826e3b07b11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:06.948145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.948118 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcce0f2-4dd8-4022-933f-e826e3b07b11-kube-api-access-gsrpt" pod "0bcce0f2-4dd8-4022-933f-e826e3b07b11" (UID: "0bcce0f2-4dd8-4022-933f-e826e3b07b11"). InnerVolumeSpecName "kube-api-access-gsrpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:06.949557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.949527 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-dshm" pod "0bcce0f2-4dd8-4022-933f-e826e3b07b11" (UID: "0bcce0f2-4dd8-4022-933f-e826e3b07b11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.949907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.949885 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-dshm" pod "fcd831ca-e923-49fb-9993-fb6954bf068b" (UID: "fcd831ca-e923-49fb-9993-fb6954bf068b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.950007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.949906 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd831ca-e923-49fb-9993-fb6954bf068b-tls-certs" pod "fcd831ca-e923-49fb-9993-fb6954bf068b" (UID: "fcd831ca-e923-49fb-9993-fb6954bf068b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:06.950071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.950035 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960f35db-200b-4e69-ae76-34f118f1af31-kube-api-access-vwknv" pod "960f35db-200b-4e69-ae76-34f118f1af31" (UID: "960f35db-200b-4e69-ae76-34f118f1af31"). InnerVolumeSpecName "kube-api-access-vwknv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:06.950135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.950066 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960f35db-200b-4e69-ae76-34f118f1af31-tls-certs" pod "960f35db-200b-4e69-ae76-34f118f1af31" (UID: "960f35db-200b-4e69-ae76-34f118f1af31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:06.950197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.950141 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-dshm" pod "960f35db-200b-4e69-ae76-34f118f1af31" (UID: "960f35db-200b-4e69-ae76-34f118f1af31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:06.951390 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:06.951358 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd831ca-e923-49fb-9993-fb6954bf068b-kube-api-access-xxd9v" pod "fcd831ca-e923-49fb-9993-fb6954bf068b" (UID: "fcd831ca-e923-49fb-9993-fb6954bf068b"). InnerVolumeSpecName "kube-api-access-xxd9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:07.046761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046728 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd831ca-e923-49fb-9993-fb6954bf068b-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046756 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046765 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046774 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxd9v\" (UniqueName: \"kubernetes.io/projected/fcd831ca-e923-49fb-9993-fb6954bf068b-kube-api-access-xxd9v\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046782 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/960f35db-200b-4e69-ae76-34f118f1af31-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046792 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046800 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046807 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bcce0f2-4dd8-4022-933f-e826e3b07b11-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046816 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsrpt\" (UniqueName: \"kubernetes.io/projected/0bcce0f2-4dd8-4022-933f-e826e3b07b11-kube-api-access-gsrpt\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046823 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwknv\" (UniqueName: \"kubernetes.io/projected/960f35db-200b-4e69-ae76-34f118f1af31-kube-api-access-vwknv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046831 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcd831ca-e923-49fb-9993-fb6954bf068b-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.046968 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.046839 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/960f35db-200b-4e69-ae76-34f118f1af31-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:07.435484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.435412 5736 generic.go:356] "Generic (PLEG): container finished" podID="960f35db-200b-4e69-ae76-34f118f1af31" containerID="552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e" exitCode=0 Oct 29 23:49:07.435484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.435460 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" Oct 29 23:49:07.436040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.435496 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" event={"ID":"960f35db-200b-4e69-ae76-34f118f1af31","Type":"ContainerDied","Data":"552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e"} Oct 29 23:49:07.436040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.435542 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv" event={"ID":"960f35db-200b-4e69-ae76-34f118f1af31","Type":"ContainerDied","Data":"7fe6dd603dd03bc18113200970f5741ce924b45c555c793093bc3ac5f8c8beeb"} Oct 29 23:49:07.436040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.435563 5736 scope.go:122] "RemoveContainer" containerID="552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e" Oct 29 23:49:07.437368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.437343 5736 generic.go:356] "Generic (PLEG): container finished" podID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerID="60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709" exitCode=0 Oct 29 23:49:07.437488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.437388 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" Oct 29 23:49:07.437488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.437432 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" event={"ID":"fcd831ca-e923-49fb-9993-fb6954bf068b","Type":"ContainerDied","Data":"60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709"} Oct 29 23:49:07.437488 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.437468 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww" event={"ID":"fcd831ca-e923-49fb-9993-fb6954bf068b","Type":"ContainerDied","Data":"df75d5a2df047df7eba59ba7733d63dc13194637711d3abbbd21e557587038bf"} Oct 29 23:49:07.439055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.439029 5736 generic.go:356] "Generic (PLEG): container finished" podID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerID="45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b" exitCode=0 Oct 29 23:49:07.439164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.439076 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" Oct 29 23:49:07.439164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.439109 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" event={"ID":"0bcce0f2-4dd8-4022-933f-e826e3b07b11","Type":"ContainerDied","Data":"45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b"} Oct 29 23:49:07.439164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.439153 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp" event={"ID":"0bcce0f2-4dd8-4022-933f-e826e3b07b11","Type":"ContainerDied","Data":"ba641eccfae93a428fc79da74f1e7d890b5a09a7e681db7e8842f253afc93702"} Oct 29 23:49:07.465716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.465692 5736 scope.go:122] "RemoveContainer" containerID="552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e" Oct 29 23:49:07.466003 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:49:07.465984 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e\": container with ID starting with 552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e not found: ID does not exist" containerID="552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e" Oct 29 23:49:07.466056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.466018 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e"} err="failed to get container status \"552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e\": rpc error: code = NotFound desc = could not find container \"552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e\": container with ID starting with 552ae8f183cd1cffd75a9f7b0b36373c9cf02b87ab09da6d10fd0c91e027a59e not found: ID does not exist" Oct 29 23:49:07.466056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.466034 5736 scope.go:122] "RemoveContainer" containerID="60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709" Oct 29 23:49:07.479052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.479031 5736 scope.go:122] "RemoveContainer" containerID="60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709" Oct 29 23:49:07.479370 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:49:07.479341 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709\": container with ID starting with 60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709 not found: ID does not exist" containerID="60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709" Oct 29 23:49:07.479465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.479379 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709"} err="failed to get container status \"60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709\": rpc error: code = NotFound desc = could not find container \"60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709\": container with ID starting with 60433eb4ad4c33623b0c445322e783a539666774688b58396699686e53d3f709 not found: ID does not exist" Oct 29 23:49:07.479465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.479412 5736 scope.go:122] "RemoveContainer" containerID="45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b" Oct 29 23:49:07.489274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.489245 5736 scope.go:122] "RemoveContainer" containerID="45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b" Oct 29 23:49:07.489534 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:49:07.489518 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b\": container with ID starting with 45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b not found: ID does not exist" containerID="45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b" Oct 29 23:49:07.489614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.489541 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b"} err="failed to get container status \"45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b\": rpc error: code = NotFound desc = could not find container \"45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b\": container with ID starting with 45440a8cf1d76c27c3ebd49c54d662412e546420065e6875a779a0fe50123f0b not found: ID does not exist" Oct 29 23:49:07.519722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.519700 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp"] Oct 29 23:49:07.523779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.523754 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/distinct-2-w0-e57216-kserve-6c47898746-qrgsp"] Oct 29 23:49:07.557913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.557890 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv"] Oct 29 23:49:07.563881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.563861 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/simulator-w0-e57216-kserve-7b88ff4f7f-5ldlv"] Oct 29 23:49:07.598675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.598653 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww"] Oct 29 23:49:07.603319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:07.603299 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e57216/premium-w0-e57216-kserve-6b9b9574c4-tbhww"] Oct 29 23:49:08.227773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.227724 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" containerID="cri-o://3e71961977144b59ea9a6169e5749c2eec9f36078cd5b6710f8be97602141dfa" gracePeriod=45 Oct 29 23:49:08.235662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.235633 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" containerID="cri-o://5f71b1edfe1200041b46f42806390cfaf1629f1ff867c90ac351256802a55135" gracePeriod=45 Oct 29 23:49:08.248976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.248910 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" containerID="cri-o://6c2774cb33fb02be2d161612a0e7380f12301d00bbd53155c432006c958bd171" gracePeriod=45 Oct 29 23:49:08.308535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.308499 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused" Oct 29 23:49:08.394746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.394701 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused" Oct 29 23:49:08.404374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.403862 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Oct 29 23:49:08.465672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.465481 5736 generic.go:356] "Generic (PLEG): container finished" podID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerID="5f71b1edfe1200041b46f42806390cfaf1629f1ff867c90ac351256802a55135" exitCode=0 Oct 29 23:49:08.466036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.465699 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" event={"ID":"e8086c08-bc06-46cf-bd5c-09f54c5b9700","Type":"ContainerDied","Data":"5f71b1edfe1200041b46f42806390cfaf1629f1ff867c90ac351256802a55135"} Oct 29 23:49:08.468258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.468063 5736 generic.go:356] "Generic (PLEG): container finished" podID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerID="6c2774cb33fb02be2d161612a0e7380f12301d00bbd53155c432006c958bd171" exitCode=0 Oct 29 23:49:08.468258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.468123 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" event={"ID":"a35990d6-4dc3-477c-83cd-14fcf9a3a84b","Type":"ContainerDied","Data":"6c2774cb33fb02be2d161612a0e7380f12301d00bbd53155c432006c958bd171"} Oct 29 23:49:08.469780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.469716 5736 generic.go:356] "Generic (PLEG): container finished" podID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerID="3e71961977144b59ea9a6169e5749c2eec9f36078cd5b6710f8be97602141dfa" exitCode=0 Oct 29 23:49:08.469780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.469754 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" event={"ID":"adbe7c2a-9f80-4dac-96b0-f8734a3759e8","Type":"ContainerDied","Data":"3e71961977144b59ea9a6169e5749c2eec9f36078cd5b6710f8be97602141dfa"} Oct 29 23:49:08.546467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.546442 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 23:49:08.574821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.574377 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tls-certs\") pod \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " Oct 29 23:49:08.574821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.574742 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tmp-dir\") pod \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " Oct 29 23:49:08.575185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.575012 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-model-cache\") pod \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " Oct 29 23:49:08.575185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.575092 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-dshm\") pod \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " Oct 29 23:49:08.575185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.575139 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-home\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-home\") pod \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " Oct 29 23:49:08.575445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.575192 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8086c08-bc06-46cf-bd5c-09f54c5b9700-kube-api-access-zt8dn\" (UniqueName: \"kubernetes.io/projected/e8086c08-bc06-46cf-bd5c-09f54c5b9700-kube-api-access-zt8dn\") pod \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\" (UID: \"e8086c08-bc06-46cf-bd5c-09f54c5b9700\") " Oct 29 23:49:08.577370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.577345 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-home" pod "e8086c08-bc06-46cf-bd5c-09f54c5b9700" (UID: "e8086c08-bc06-46cf-bd5c-09f54c5b9700"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.577620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.577562 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tmp-dir" pod "e8086c08-bc06-46cf-bd5c-09f54c5b9700" (UID: "e8086c08-bc06-46cf-bd5c-09f54c5b9700"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.577720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.577608 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-model-cache" pod "e8086c08-bc06-46cf-bd5c-09f54c5b9700" (UID: "e8086c08-bc06-46cf-bd5c-09f54c5b9700"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.579830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.579800 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tls-certs" pod "e8086c08-bc06-46cf-bd5c-09f54c5b9700" (UID: "e8086c08-bc06-46cf-bd5c-09f54c5b9700"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:08.580507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.580480 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8086c08-bc06-46cf-bd5c-09f54c5b9700-kube-api-access-zt8dn" pod "e8086c08-bc06-46cf-bd5c-09f54c5b9700" (UID: "e8086c08-bc06-46cf-bd5c-09f54c5b9700"). InnerVolumeSpecName "kube-api-access-zt8dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:08.580791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.580764 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-dshm" pod "e8086c08-bc06-46cf-bd5c-09f54c5b9700" (UID: "e8086c08-bc06-46cf-bd5c-09f54c5b9700"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.611086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.611032 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:49:08.614493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.614467 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:49:08.677432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.677399 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-kube-api-access-vqqd9\" (UniqueName: \"kubernetes.io/projected/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-kube-api-access-vqqd9\") pod \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " Oct 29 23:49:08.677684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.677444 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-home\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-home\") pod \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " Oct 29 23:49:08.677684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.677641 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-dshm\") pod \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " Oct 29 23:49:08.677684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.677676 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-model-cache\") pod \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " Oct 29 23:49:08.677869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.677726 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-model-cache\") pod \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " Oct 29 23:49:08.677869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.677744 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-home" pod "adbe7c2a-9f80-4dac-96b0-f8734a3759e8" (UID: "adbe7c2a-9f80-4dac-96b0-f8734a3759e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.677869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.677845 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tls-certs\") pod \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " Oct 29 23:49:08.678049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.677899 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-kube-api-access-229kf\" (UniqueName: \"kubernetes.io/projected/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-kube-api-access-229kf\") pod \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " Oct 29 23:49:08.678049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.677943 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-dshm\") pod \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " Oct 29 23:49:08.678049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.678023 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-home\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-home\") pod \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " Oct 29 23:49:08.678248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.678222 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-model-cache" pod "a35990d6-4dc3-477c-83cd-14fcf9a3a84b" (UID: "a35990d6-4dc3-477c-83cd-14fcf9a3a84b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.678434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.678411 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-model-cache" pod "adbe7c2a-9f80-4dac-96b0-f8734a3759e8" (UID: "adbe7c2a-9f80-4dac-96b0-f8734a3759e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.680368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.680327 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-home" pod "a35990d6-4dc3-477c-83cd-14fcf9a3a84b" (UID: "a35990d6-4dc3-477c-83cd-14fcf9a3a84b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.680478 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.680418 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tmp-dir\") pod \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " Oct 29 23:49:08.681108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.680770 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tmp-dir\") pod \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\" (UID: \"adbe7c2a-9f80-4dac-96b0-f8734a3759e8\") " Oct 29 23:49:08.681108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.680810 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tmp-dir" pod "adbe7c2a-9f80-4dac-96b0-f8734a3759e8" (UID: "adbe7c2a-9f80-4dac-96b0-f8734a3759e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.681108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.680837 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tls-certs\") pod \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\" (UID: \"a35990d6-4dc3-477c-83cd-14fcf9a3a84b\") " Oct 29 23:49:08.681630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681607 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681634 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681647 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681660 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681674 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681686 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681697 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681708 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681718 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8086c08-bc06-46cf-bd5c-09f54c5b9700-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681726 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt8dn\" (UniqueName: \"kubernetes.io/projected/e8086c08-bc06-46cf-bd5c-09f54c5b9700-kube-api-access-zt8dn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.681736 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.681735 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.682324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.682302 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tmp-dir" pod "a35990d6-4dc3-477c-83cd-14fcf9a3a84b" (UID: "a35990d6-4dc3-477c-83cd-14fcf9a3a84b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.682426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.682332 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-kube-api-access-vqqd9" pod "a35990d6-4dc3-477c-83cd-14fcf9a3a84b" (UID: "a35990d6-4dc3-477c-83cd-14fcf9a3a84b"). InnerVolumeSpecName "kube-api-access-vqqd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:08.686929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.686888 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-dshm" pod "a35990d6-4dc3-477c-83cd-14fcf9a3a84b" (UID: "a35990d6-4dc3-477c-83cd-14fcf9a3a84b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.687458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.687422 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tls-certs" pod "adbe7c2a-9f80-4dac-96b0-f8734a3759e8" (UID: "adbe7c2a-9f80-4dac-96b0-f8734a3759e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:08.687557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.687453 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-dshm" pod "adbe7c2a-9f80-4dac-96b0-f8734a3759e8" (UID: "adbe7c2a-9f80-4dac-96b0-f8734a3759e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:49:08.687898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.687864 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-kube-api-access-229kf" pod "adbe7c2a-9f80-4dac-96b0-f8734a3759e8" (UID: "adbe7c2a-9f80-4dac-96b0-f8734a3759e8"). InnerVolumeSpecName "kube-api-access-229kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:08.689284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.689255 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tls-certs" pod "a35990d6-4dc3-477c-83cd-14fcf9a3a84b" (UID: "a35990d6-4dc3-477c-83cd-14fcf9a3a84b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:08.783499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.783419 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.783499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.783455 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-229kf\" (UniqueName: \"kubernetes.io/projected/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-kube-api-access-229kf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.783499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.783465 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbe7c2a-9f80-4dac-96b0-f8734a3759e8-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.783499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.783474 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.783499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.783484 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.783811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.783507 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqd9\" (UniqueName: \"kubernetes.io/projected/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-kube-api-access-vqqd9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:08.783811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:08.783515 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35990d6-4dc3-477c-83cd-14fcf9a3a84b-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:09.005852 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.005808 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" path="/var/lib/kubelet/pods/0bcce0f2-4dd8-4022-933f-e826e3b07b11/volumes" Oct 29 23:49:09.006377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.006352 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960f35db-200b-4e69-ae76-34f118f1af31" path="/var/lib/kubelet/pods/960f35db-200b-4e69-ae76-34f118f1af31/volumes" Oct 29 23:49:09.006884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.006864 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" path="/var/lib/kubelet/pods/fcd831ca-e923-49fb-9993-fb6954bf068b/volumes" Oct 29 23:49:09.480679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.480643 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" Oct 29 23:49:09.480679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.480650 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz" event={"ID":"e8086c08-bc06-46cf-bd5c-09f54c5b9700","Type":"ContainerDied","Data":"9eb0d0fe1a7b56f25ecfaa31e35eec2c326fbb13ddefc5ef3dc0d766b6765211"} Oct 29 23:49:09.481170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.480701 5736 scope.go:122] "RemoveContainer" containerID="5f71b1edfe1200041b46f42806390cfaf1629f1ff867c90ac351256802a55135" Oct 29 23:49:09.482331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.482310 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" Oct 29 23:49:09.482481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.482340 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw" event={"ID":"a35990d6-4dc3-477c-83cd-14fcf9a3a84b","Type":"ContainerDied","Data":"2eeb9f3199cbad31975e60d5e1e71248d6e129e0857da432194fef2ae2093db7"} Oct 29 23:49:09.483955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.483936 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" Oct 29 23:49:09.484079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.483986 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh" event={"ID":"adbe7c2a-9f80-4dac-96b0-f8734a3759e8","Type":"ContainerDied","Data":"b43be0b03dc7a6bf07464e8f1288c349f821dd59b021e2a1bb9ce0eaf60e9eb3"} Oct 29 23:49:09.493155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.493138 5736 scope.go:122] "RemoveContainer" containerID="6c2774cb33fb02be2d161612a0e7380f12301d00bbd53155c432006c958bd171" Oct 29 23:49:09.502643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.502623 5736 scope.go:122] "RemoveContainer" containerID="3e71961977144b59ea9a6169e5749c2eec9f36078cd5b6710f8be97602141dfa" Oct 29 23:49:09.534164 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.534138 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh"] Oct 29 23:49:09.545857 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.545832 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-2-w4-ed8218-kserve-69ffb9d49b-k8fnh"] Oct 29 23:49:09.580032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.580007 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz"] Oct 29 23:49:09.586197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.586174 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/distinct-w4-ed8218-kserve-679d796d67-km4zz"] Oct 29 23:49:09.619789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.619767 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw"] Oct 29 23:49:09.624904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:09.624882 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/simulator-w4-ed8218-kserve-56f9cffdb5-rhrrw"] Oct 29 23:49:10.204837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.204803 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn"] Oct 29 23:49:10.205455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205436 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" Oct 29 23:49:10.205542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205457 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" Oct 29 23:49:10.205542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205474 5736 cpu_manager.go:414] "containerMap: removing container" podUID="af2d7af5-e65f-47c1-9323-9851b1721066" containerName="revoke-keys" Oct 29 23:49:10.205542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205482 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2d7af5-e65f-47c1-9323-9851b1721066" containerName="revoke-keys" Oct 29 23:49:10.205542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205511 5736 cpu_manager.go:414] "containerMap: removing container" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" Oct 29 23:49:10.205542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205516 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" Oct 29 23:49:10.205542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205528 5736 cpu_manager.go:414] "containerMap: removing container" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" Oct 29 23:49:10.205542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205533 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205553 5736 cpu_manager.go:414] "containerMap: removing container" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205558 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205591 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205598 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205611 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205616 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205716 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205728 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205736 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205742 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd831ca-e923-49fb-9993-fb6954bf068b" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205748 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2d7af5-e65f-47c1-9323-9851b1721066" containerName="revoke-keys" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205758 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcce0f2-4dd8-4022-933f-e826e3b07b11" containerName="main" Oct 29 23:49:10.205804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.205765 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="960f35db-200b-4e69-ae76-34f118f1af31" containerName="main" Oct 29 23:49:10.210560 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.210541 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.230740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.230718 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:49:10.230897 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.230812 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:49:10.231567 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.231544 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss"] Oct 29 23:49:10.236103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.236081 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn"] Oct 29 23:49:10.236228 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.236188 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.261679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.261650 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss"] Oct 29 23:49:10.302737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.302695 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f3e908-qv5kn\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.302901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.302806 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnqgn\" (UniqueName: \"kubernetes.io/projected/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-kube-api-access-cnqgn\") pod \"maas-api-revoke-sub-85f3e908-qv5kn\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.302995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.302901 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ab28727-q82ss\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.302995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.302954 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f3e908-qv5kn\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.303122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.303002 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ab28727-q82ss\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.303122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.303031 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85977\" (UniqueName: \"kubernetes.io/projected/0a6bca83-635f-49f1-8485-0033968f6318-kube-api-access-85977\") pod \"maas-api-revoke-sub-6ab28727-q82ss\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.404333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.404298 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f3e908-qv5kn\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.404516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.404374 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnqgn\" (UniqueName: \"kubernetes.io/projected/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-kube-api-access-cnqgn\") pod \"maas-api-revoke-sub-85f3e908-qv5kn\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.404516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.404433 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ab28727-q82ss\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.404516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.404490 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f3e908-qv5kn\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.404727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.404516 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ab28727-q82ss\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.404727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.404546 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85977\" (UniqueName: \"kubernetes.io/projected/0a6bca83-635f-49f1-8485-0033968f6318-kube-api-access-85977\") pod \"maas-api-revoke-sub-6ab28727-q82ss\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.405152 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.405126 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85f3e908-qv5kn\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.405805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.405774 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ab28727-q82ss\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.407671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.407643 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85f3e908-qv5kn\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.407861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.407845 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ab28727-q82ss\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.428344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.428312 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85977\" (UniqueName: \"kubernetes.io/projected/0a6bca83-635f-49f1-8485-0033968f6318-kube-api-access-85977\") pod \"maas-api-revoke-sub-6ab28727-q82ss\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.428562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.428538 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnqgn\" (UniqueName: \"kubernetes.io/projected/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-kube-api-access-cnqgn\") pod \"maas-api-revoke-sub-85f3e908-qv5kn\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.525033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.524990 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:10.553053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.553027 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:10.877525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.877447 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn"] Oct 29 23:49:10.881792 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:49:10.881762 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e8aa16_e4ee_4c0c_ae66_f547b1fcc2d1.slice/crio-c911362a313f37b6b83897ef5d3f96263a1caa7f29a4bc76ada1a9b804b30882 WatchSource:0}: Error finding container c911362a313f37b6b83897ef5d3f96263a1caa7f29a4bc76ada1a9b804b30882: Status 404 returned error can't find the container with id c911362a313f37b6b83897ef5d3f96263a1caa7f29a4bc76ada1a9b804b30882 Oct 29 23:49:10.893417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:10.893389 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss"] Oct 29 23:49:10.896391 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:49:10.896363 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6bca83_635f_49f1_8485_0033968f6318.slice/crio-9b50cbde2bf51968482e676bb2b910e45d2ee8cc1cb4a759af25e65943cdfe65 WatchSource:0}: Error finding container 9b50cbde2bf51968482e676bb2b910e45d2ee8cc1cb4a759af25e65943cdfe65: Status 404 returned error can't find the container with id 9b50cbde2bf51968482e676bb2b910e45d2ee8cc1cb4a759af25e65943cdfe65 Oct 29 23:49:11.006318 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:11.006289 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a35990d6-4dc3-477c-83cd-14fcf9a3a84b" path="/var/lib/kubelet/pods/a35990d6-4dc3-477c-83cd-14fcf9a3a84b/volumes" Oct 29 23:49:11.006799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:11.006784 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbe7c2a-9f80-4dac-96b0-f8734a3759e8" path="/var/lib/kubelet/pods/adbe7c2a-9f80-4dac-96b0-f8734a3759e8/volumes" Oct 29 23:49:11.007229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:11.007214 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8086c08-bc06-46cf-bd5c-09f54c5b9700" path="/var/lib/kubelet/pods/e8086c08-bc06-46cf-bd5c-09f54c5b9700/volumes" Oct 29 23:49:11.505445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:11.505405 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" event={"ID":"0a6bca83-635f-49f1-8485-0033968f6318","Type":"ContainerStarted","Data":"542fb67b86d69eda7ad0840258d82bcc5337d75e56c2ea2ed3ad39aa4e0f3d9c"} Oct 29 23:49:11.505663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:11.505469 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" event={"ID":"0a6bca83-635f-49f1-8485-0033968f6318","Type":"ContainerStarted","Data":"9b50cbde2bf51968482e676bb2b910e45d2ee8cc1cb4a759af25e65943cdfe65"} Oct 29 23:49:11.507132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:11.507103 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" event={"ID":"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1","Type":"ContainerStarted","Data":"c84c7418664f41733bcb4921dd2e2370510cc6be92b8134215fa8fc9bc5d691d"} Oct 29 23:49:11.507236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:11.507145 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" event={"ID":"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1","Type":"ContainerStarted","Data":"c911362a313f37b6b83897ef5d3f96263a1caa7f29a4bc76ada1a9b804b30882"} Oct 29 23:49:11.549167 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:11.549090 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" podStartSLOduration=1.5490736950000001 podStartE2EDuration="1.549073695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:49:11.547345104 +0000 UTC m=+3948.663264710" watchObservedRunningTime="2026-10-29 23:49:11.549073695 +0000 UTC m=+3948.664993302" Oct 29 23:49:11.591628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:11.591563 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" podStartSLOduration=1.5915509810000001 podStartE2EDuration="1.591550981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:49:11.587898371 +0000 UTC m=+3948.703817977" watchObservedRunningTime="2026-10-29 23:49:11.591550981 +0000 UTC m=+3948.707470584" Oct 29 23:49:12.540106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:12.539806 5736 generic.go:356] "Generic (PLEG): container finished" podID="0a6bca83-635f-49f1-8485-0033968f6318" containerID="542fb67b86d69eda7ad0840258d82bcc5337d75e56c2ea2ed3ad39aa4e0f3d9c" exitCode=0 Oct 29 23:49:12.540106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:12.539966 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" event={"ID":"0a6bca83-635f-49f1-8485-0033968f6318","Type":"ContainerDied","Data":"542fb67b86d69eda7ad0840258d82bcc5337d75e56c2ea2ed3ad39aa4e0f3d9c"} Oct 29 23:49:12.543627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:12.543563 5736 generic.go:356] "Generic (PLEG): container finished" podID="15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1" containerID="c84c7418664f41733bcb4921dd2e2370510cc6be92b8134215fa8fc9bc5d691d" exitCode=0 Oct 29 23:49:12.543791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:12.543615 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" event={"ID":"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1","Type":"ContainerDied","Data":"c84c7418664f41733bcb4921dd2e2370510cc6be92b8134215fa8fc9bc5d691d"} Oct 29 23:49:13.727068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.727010 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:13.761611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.761549 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:13.867530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.867458 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-openshift-ca\") pod \"0a6bca83-635f-49f1-8485-0033968f6318\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " Oct 29 23:49:13.867530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.867502 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-openshift-ca\") pod \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " Oct 29 23:49:13.867746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.867542 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-xks-ca\") pod \"0a6bca83-635f-49f1-8485-0033968f6318\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " Oct 29 23:49:13.867746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.867661 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-kube-api-access-cnqgn\" (UniqueName: \"kubernetes.io/projected/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-kube-api-access-cnqgn\") pod \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " Oct 29 23:49:13.867746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.867740 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6bca83-635f-49f1-8485-0033968f6318-kube-api-access-85977\" (UniqueName: \"kubernetes.io/projected/0a6bca83-635f-49f1-8485-0033968f6318-kube-api-access-85977\") pod \"0a6bca83-635f-49f1-8485-0033968f6318\" (UID: \"0a6bca83-635f-49f1-8485-0033968f6318\") " Oct 29 23:49:13.867878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.867772 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-xks-ca\") pod \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\" (UID: \"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1\") " Oct 29 23:49:13.867946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.867865 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-openshift-ca" pod "0a6bca83-635f-49f1-8485-0033968f6318" (UID: "0a6bca83-635f-49f1-8485-0033968f6318"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:13.868004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.867936 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-openshift-ca" pod "15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1" (UID: "15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:13.868652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.868621 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:13.868795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.868655 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:13.869906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.869874 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-xks-ca" pod "0a6bca83-635f-49f1-8485-0033968f6318" (UID: "0a6bca83-635f-49f1-8485-0033968f6318"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:13.870047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.870008 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6bca83-635f-49f1-8485-0033968f6318-kube-api-access-85977" pod "0a6bca83-635f-49f1-8485-0033968f6318" (UID: "0a6bca83-635f-49f1-8485-0033968f6318"). InnerVolumeSpecName "kube-api-access-85977". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:13.870119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.870042 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-xks-ca" pod "15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1" (UID: "15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:13.870119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.870077 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-kube-api-access-cnqgn" pod "15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1" (UID: "15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1"). InnerVolumeSpecName "kube-api-access-cnqgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:13.969718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.969681 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnqgn\" (UniqueName: \"kubernetes.io/projected/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-kube-api-access-cnqgn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:13.969718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.969717 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85977\" (UniqueName: \"kubernetes.io/projected/0a6bca83-635f-49f1-8485-0033968f6318-kube-api-access-85977\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:13.969718 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.969727 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:13.969903 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:13.969736 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6bca83-635f-49f1-8485-0033968f6318-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:14.560371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:14.560336 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" Oct 29 23:49:14.560371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:14.560365 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss" event={"ID":"0a6bca83-635f-49f1-8485-0033968f6318","Type":"ContainerDied","Data":"9b50cbde2bf51968482e676bb2b910e45d2ee8cc1cb4a759af25e65943cdfe65"} Oct 29 23:49:14.560650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:14.560414 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b50cbde2bf51968482e676bb2b910e45d2ee8cc1cb4a759af25e65943cdfe65" Oct 29 23:49:14.562065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:14.562044 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" Oct 29 23:49:14.562065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:14.562048 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn" event={"ID":"15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1","Type":"ContainerDied","Data":"c911362a313f37b6b83897ef5d3f96263a1caa7f29a4bc76ada1a9b804b30882"} Oct 29 23:49:14.562215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:14.562089 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c911362a313f37b6b83897ef5d3f96263a1caa7f29a4bc76ada1a9b804b30882" Oct 29 23:49:18.891465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:18.891422 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h"] Oct 29 23:49:18.898268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:18.898238 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-26b322-pfv4h"] Oct 29 23:49:19.004503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:19.004469 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa8431e-977e-49bd-ada8-a617a8f8122f" path="/var/lib/kubelet/pods/7aa8431e-977e-49bd-ada8-a617a8f8122f/volumes" Oct 29 23:49:26.824259 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:49:26.824211 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:49:28.354148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.354109 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs"] Oct 29 23:49:28.355003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.354977 5736 cpu_manager.go:414] "containerMap: removing container" podUID="15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1" containerName="revoke-keys" Oct 29 23:49:28.355113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.355003 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1" containerName="revoke-keys" Oct 29 23:49:28.355113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.355030 5736 cpu_manager.go:414] "containerMap: removing container" podUID="0a6bca83-635f-49f1-8485-0033968f6318" containerName="revoke-keys" Oct 29 23:49:28.355113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.355039 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6bca83-635f-49f1-8485-0033968f6318" containerName="revoke-keys" Oct 29 23:49:28.355285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.355225 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1" containerName="revoke-keys" Oct 29 23:49:28.355285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.355242 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6bca83-635f-49f1-8485-0033968f6318" containerName="revoke-keys" Oct 29 23:49:28.360335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.360312 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.381470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.381447 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:49:28.393851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.393826 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:49:28.398730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.398709 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs"] Oct 29 23:49:28.485107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.485072 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.485306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.485116 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gp6j\" (UniqueName: \"kubernetes.io/projected/279f8c23-d226-4604-bdca-bf0fe8c1164c-kube-api-access-5gp6j\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.485306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.485245 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.587004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.586964 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.587197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.587016 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gp6j\" (UniqueName: \"kubernetes.io/projected/279f8c23-d226-4604-bdca-bf0fe8c1164c-kube-api-access-5gp6j\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.587197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.587071 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.587724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.587697 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.589512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.589488 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.617082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.617020 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gp6j\" (UniqueName: \"kubernetes.io/projected/279f8c23-d226-4604-bdca-bf0fe8c1164c-kube-api-access-5gp6j\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:28.675273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:28.675241 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:29.008063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:29.008034 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs"] Oct 29 23:49:29.011460 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:49:29.011428 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279f8c23_d226_4604_bdca_bf0fe8c1164c.slice/crio-5c6536b87e2d8dbaaf7c3f66f9a72b4753985ad6485660e379d87616a738532c WatchSource:0}: Error finding container 5c6536b87e2d8dbaaf7c3f66f9a72b4753985ad6485660e379d87616a738532c: Status 404 returned error can't find the container with id 5c6536b87e2d8dbaaf7c3f66f9a72b4753985ad6485660e379d87616a738532c Oct 29 23:49:29.691864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:29.691778 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" event={"ID":"279f8c23-d226-4604-bdca-bf0fe8c1164c","Type":"ContainerStarted","Data":"dbb1bd8160825681eba73d5941685ef34ce07ff19cac9a1b9accf23aecc57d55"} Oct 29 23:49:29.691864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:29.691839 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" event={"ID":"279f8c23-d226-4604-bdca-bf0fe8c1164c","Type":"ContainerStarted","Data":"5c6536b87e2d8dbaaf7c3f66f9a72b4753985ad6485660e379d87616a738532c"} Oct 29 23:49:30.701726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:30.701684 5736 generic.go:356] "Generic (PLEG): container finished" podID="279f8c23-d226-4604-bdca-bf0fe8c1164c" containerID="dbb1bd8160825681eba73d5941685ef34ce07ff19cac9a1b9accf23aecc57d55" exitCode=0 Oct 29 23:49:30.702144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:30.701750 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" event={"ID":"279f8c23-d226-4604-bdca-bf0fe8c1164c","Type":"ContainerDied","Data":"dbb1bd8160825681eba73d5941685ef34ce07ff19cac9a1b9accf23aecc57d55"} Oct 29 23:49:31.173050 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.173017 5736 scope.go:122] "RemoveContainer" containerID="92407076100e03b3c11600243d977129dd260723afb148af2d5a29eff577c61a" Oct 29 23:49:31.184024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.183999 5736 scope.go:122] "RemoveContainer" containerID="9076fa0cc96a70f144079fcd1aeb0258379f5813c94f4a41c1044e3bbabb7484" Oct 29 23:49:31.193790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.193761 5736 scope.go:122] "RemoveContainer" containerID="b29ec2d62cf732738ba1ec0c378234026ae84fe7b6fdba31b80cfb30dacbca9d" Oct 29 23:49:31.202974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.202957 5736 scope.go:122] "RemoveContainer" containerID="651fa3be849b00b78155313178f783ef5c04d9c001ad340c85d8176120dc9d8b" Oct 29 23:49:31.890674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.890638 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:31.939944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.939910 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-xks-ca\") pod \"279f8c23-d226-4604-bdca-bf0fe8c1164c\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " Oct 29 23:49:31.940103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.939981 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279f8c23-d226-4604-bdca-bf0fe8c1164c-kube-api-access-5gp6j\" (UniqueName: \"kubernetes.io/projected/279f8c23-d226-4604-bdca-bf0fe8c1164c-kube-api-access-5gp6j\") pod \"279f8c23-d226-4604-bdca-bf0fe8c1164c\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " Oct 29 23:49:31.940103 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.940078 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-openshift-ca\") pod \"279f8c23-d226-4604-bdca-bf0fe8c1164c\" (UID: \"279f8c23-d226-4604-bdca-bf0fe8c1164c\") " Oct 29 23:49:31.940528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.940488 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-openshift-ca" pod "279f8c23-d226-4604-bdca-bf0fe8c1164c" (UID: "279f8c23-d226-4604-bdca-bf0fe8c1164c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:31.940815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.940791 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:31.942196 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.942161 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279f8c23-d226-4604-bdca-bf0fe8c1164c-kube-api-access-5gp6j" pod "279f8c23-d226-4604-bdca-bf0fe8c1164c" (UID: "279f8c23-d226-4604-bdca-bf0fe8c1164c"). InnerVolumeSpecName "kube-api-access-5gp6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:31.942497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:31.942473 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-xks-ca" pod "279f8c23-d226-4604-bdca-bf0fe8c1164c" (UID: "279f8c23-d226-4604-bdca-bf0fe8c1164c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:32.042173 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:32.042085 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279f8c23-d226-4604-bdca-bf0fe8c1164c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:32.042173 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:32.042129 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gp6j\" (UniqueName: \"kubernetes.io/projected/279f8c23-d226-4604-bdca-bf0fe8c1164c-kube-api-access-5gp6j\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:32.721670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:32.721637 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" Oct 29 23:49:32.721842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:32.721669 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs" event={"ID":"279f8c23-d226-4604-bdca-bf0fe8c1164c","Type":"ContainerDied","Data":"5c6536b87e2d8dbaaf7c3f66f9a72b4753985ad6485660e379d87616a738532c"} Oct 29 23:49:32.721842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:32.721718 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c6536b87e2d8dbaaf7c3f66f9a72b4753985ad6485660e379d87616a738532c" Oct 29 23:49:33.063115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:33.063031 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7"] Oct 29 23:49:33.073928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:33.073896 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb002-zw8x7"] Oct 29 23:49:34.471917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.471094 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8"] Oct 29 23:49:34.476653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.472198 5736 cpu_manager.go:414] "containerMap: removing container" podUID="279f8c23-d226-4604-bdca-bf0fe8c1164c" containerName="revoke-keys" Oct 29 23:49:34.476653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.472225 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279f8c23-d226-4604-bdca-bf0fe8c1164c" containerName="revoke-keys" Oct 29 23:49:34.476653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.472642 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="279f8c23-d226-4604-bdca-bf0fe8c1164c" containerName="revoke-keys" Oct 29 23:49:34.493832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.493797 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55"] Oct 29 23:49:34.509559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.494110 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.529306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.527179 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:49:34.529306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.527253 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:49:34.563539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.558116 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8"] Oct 29 23:49:34.563539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.558251 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.567398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.567365 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55"] Oct 29 23:49:34.647461 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.647423 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-790f2056-l9b55\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.647688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.647502 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bgx\" (UniqueName: \"kubernetes.io/projected/1077baad-3485-4973-951f-50e8ff9d5c0e-kube-api-access-q8bgx\") pod \"maas-api-revoke-sub-e4850bd6-wdkl8\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.647688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.647666 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4850bd6-wdkl8\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.647833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.647757 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm294\" (UniqueName: \"kubernetes.io/projected/bedc11e4-3f71-4390-a308-564524688cdd-kube-api-access-fm294\") pod \"maas-api-revoke-sub-790f2056-l9b55\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.647895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.647869 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-790f2056-l9b55\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.648127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.648105 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4850bd6-wdkl8\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.750442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.750348 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-790f2056-l9b55\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.750737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.750710 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bgx\" (UniqueName: \"kubernetes.io/projected/1077baad-3485-4973-951f-50e8ff9d5c0e-kube-api-access-q8bgx\") pod \"maas-api-revoke-sub-e4850bd6-wdkl8\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.750914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.750896 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4850bd6-wdkl8\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.756607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.751069 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm294\" (UniqueName: \"kubernetes.io/projected/bedc11e4-3f71-4390-a308-564524688cdd-kube-api-access-fm294\") pod \"maas-api-revoke-sub-790f2056-l9b55\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.756887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.756860 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-790f2056-l9b55\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.757273 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.757249 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4850bd6-wdkl8\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.762604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.758766 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4850bd6-wdkl8\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.824364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.768681 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-790f2056-l9b55\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.824656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.815000 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4850bd6-wdkl8\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.824811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.815286 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-790f2056-l9b55\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.824952 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.820167 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm294\" (UniqueName: \"kubernetes.io/projected/bedc11e4-3f71-4390-a308-564524688cdd-kube-api-access-fm294\") pod \"maas-api-revoke-sub-790f2056-l9b55\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:34.825080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.820812 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bgx\" (UniqueName: \"kubernetes.io/projected/1077baad-3485-4973-951f-50e8ff9d5c0e-kube-api-access-q8bgx\") pod \"maas-api-revoke-sub-e4850bd6-wdkl8\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.865250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.865216 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:34.885688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:34.885648 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:35.015982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:35.015907 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70dcfde7-dc28-4c55-890d-ac822e0886a2" path="/var/lib/kubelet/pods/70dcfde7-dc28-4c55-890d-ac822e0886a2/volumes" Oct 29 23:49:35.200896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:35.199306 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8"] Oct 29 23:49:35.203821 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:49:35.203787 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1077baad_3485_4973_951f_50e8ff9d5c0e.slice/crio-baa1ab7b0fa9476ed97d083c81e0245d51f0734d8a6614cfe3db1315b660a8c3 WatchSource:0}: Error finding container baa1ab7b0fa9476ed97d083c81e0245d51f0734d8a6614cfe3db1315b660a8c3: Status 404 returned error can't find the container with id baa1ab7b0fa9476ed97d083c81e0245d51f0734d8a6614cfe3db1315b660a8c3 Oct 29 23:49:35.266886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:35.266860 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55"] Oct 29 23:49:35.268882 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:49:35.268852 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedc11e4_3f71_4390_a308_564524688cdd.slice/crio-1205ebd05bf42ab4a011b73fe46cc2ae6dfdfead621b0009c2ea1cbd4a430acd WatchSource:0}: Error finding container 1205ebd05bf42ab4a011b73fe46cc2ae6dfdfead621b0009c2ea1cbd4a430acd: Status 404 returned error can't find the container with id 1205ebd05bf42ab4a011b73fe46cc2ae6dfdfead621b0009c2ea1cbd4a430acd Oct 29 23:49:35.839434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:35.839347 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" event={"ID":"1077baad-3485-4973-951f-50e8ff9d5c0e","Type":"ContainerStarted","Data":"9e15d4990d9ee713058fb6d709924a946bc0ec590709532b8181a321a20ad2a2"} Oct 29 23:49:35.839906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:35.839438 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" event={"ID":"1077baad-3485-4973-951f-50e8ff9d5c0e","Type":"ContainerStarted","Data":"baa1ab7b0fa9476ed97d083c81e0245d51f0734d8a6614cfe3db1315b660a8c3"} Oct 29 23:49:35.841060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:35.841031 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" event={"ID":"bedc11e4-3f71-4390-a308-564524688cdd","Type":"ContainerStarted","Data":"98bc183e98780b5426e732780c4782655e7511603da563d5061146c97b18ee88"} Oct 29 23:49:35.841191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:35.841078 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" event={"ID":"bedc11e4-3f71-4390-a308-564524688cdd","Type":"ContainerStarted","Data":"1205ebd05bf42ab4a011b73fe46cc2ae6dfdfead621b0009c2ea1cbd4a430acd"} Oct 29 23:49:35.906761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:35.906714 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" podStartSLOduration=1.906701465 podStartE2EDuration="1.906701465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:49:35.903056153 +0000 UTC m=+3973.018975759" watchObservedRunningTime="2026-10-29 23:49:35.906701465 +0000 UTC m=+3973.022621071" Oct 29 23:49:35.966055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:35.966000 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" podStartSLOduration=1.965983778 podStartE2EDuration="1.965983778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:49:35.962410161 +0000 UTC m=+3973.078329766" watchObservedRunningTime="2026-10-29 23:49:35.965983778 +0000 UTC m=+3973.081903383" Oct 29 23:49:36.851313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:36.851280 5736 generic.go:356] "Generic (PLEG): container finished" podID="1077baad-3485-4973-951f-50e8ff9d5c0e" containerID="9e15d4990d9ee713058fb6d709924a946bc0ec590709532b8181a321a20ad2a2" exitCode=0 Oct 29 23:49:36.851752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:36.851366 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" event={"ID":"1077baad-3485-4973-951f-50e8ff9d5c0e","Type":"ContainerDied","Data":"9e15d4990d9ee713058fb6d709924a946bc0ec590709532b8181a321a20ad2a2"} Oct 29 23:49:36.853223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:36.853192 5736 generic.go:356] "Generic (PLEG): container finished" podID="bedc11e4-3f71-4390-a308-564524688cdd" containerID="98bc183e98780b5426e732780c4782655e7511603da563d5061146c97b18ee88" exitCode=0 Oct 29 23:49:36.853369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:36.853229 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" event={"ID":"bedc11e4-3f71-4390-a308-564524688cdd","Type":"ContainerDied","Data":"98bc183e98780b5426e732780c4782655e7511603da563d5061146c97b18ee88"} Oct 29 23:49:38.033348 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.033313 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:38.068934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.068909 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:38.129742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.129657 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-openshift-ca\") pod \"bedc11e4-3f71-4390-a308-564524688cdd\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " Oct 29 23:49:38.129907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.129745 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-xks-ca\") pod \"1077baad-3485-4973-951f-50e8ff9d5c0e\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " Oct 29 23:49:38.129907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.129896 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1077baad-3485-4973-951f-50e8ff9d5c0e-kube-api-access-q8bgx\" (UniqueName: \"kubernetes.io/projected/1077baad-3485-4973-951f-50e8ff9d5c0e-kube-api-access-q8bgx\") pod \"1077baad-3485-4973-951f-50e8ff9d5c0e\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " Oct 29 23:49:38.130084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.129996 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-xks-ca\") pod \"bedc11e4-3f71-4390-a308-564524688cdd\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " Oct 29 23:49:38.130147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.130104 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-openshift-ca" pod "bedc11e4-3f71-4390-a308-564524688cdd" (UID: "bedc11e4-3f71-4390-a308-564524688cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:38.130147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.130118 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedc11e4-3f71-4390-a308-564524688cdd-kube-api-access-fm294\" (UniqueName: \"kubernetes.io/projected/bedc11e4-3f71-4390-a308-564524688cdd-kube-api-access-fm294\") pod \"bedc11e4-3f71-4390-a308-564524688cdd\" (UID: \"bedc11e4-3f71-4390-a308-564524688cdd\") " Oct 29 23:49:38.130262 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.130201 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-openshift-ca\") pod \"1077baad-3485-4973-951f-50e8ff9d5c0e\" (UID: \"1077baad-3485-4973-951f-50e8ff9d5c0e\") " Oct 29 23:49:38.130901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.130863 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-openshift-ca" pod "1077baad-3485-4973-951f-50e8ff9d5c0e" (UID: "1077baad-3485-4973-951f-50e8ff9d5c0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:38.131098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.131071 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:38.131193 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.131103 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:38.133911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.133885 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-xks-ca" pod "1077baad-3485-4973-951f-50e8ff9d5c0e" (UID: "1077baad-3485-4973-951f-50e8ff9d5c0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:38.134024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.133911 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1077baad-3485-4973-951f-50e8ff9d5c0e-kube-api-access-q8bgx" pod "1077baad-3485-4973-951f-50e8ff9d5c0e" (UID: "1077baad-3485-4973-951f-50e8ff9d5c0e"). InnerVolumeSpecName "kube-api-access-q8bgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:38.134024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.133968 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedc11e4-3f71-4390-a308-564524688cdd-kube-api-access-fm294" pod "bedc11e4-3f71-4390-a308-564524688cdd" (UID: "bedc11e4-3f71-4390-a308-564524688cdd"). InnerVolumeSpecName "kube-api-access-fm294". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:38.135090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.135054 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-xks-ca" pod "bedc11e4-3f71-4390-a308-564524688cdd" (UID: "bedc11e4-3f71-4390-a308-564524688cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:38.232882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.232847 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1077baad-3485-4973-951f-50e8ff9d5c0e-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:38.232882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.232877 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8bgx\" (UniqueName: \"kubernetes.io/projected/1077baad-3485-4973-951f-50e8ff9d5c0e-kube-api-access-q8bgx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:38.232882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.232890 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bedc11e4-3f71-4390-a308-564524688cdd-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:38.233113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.232899 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm294\" (UniqueName: \"kubernetes.io/projected/bedc11e4-3f71-4390-a308-564524688cdd-kube-api-access-fm294\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:38.871310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.871274 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" Oct 29 23:49:38.871491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.871305 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8" event={"ID":"1077baad-3485-4973-951f-50e8ff9d5c0e","Type":"ContainerDied","Data":"baa1ab7b0fa9476ed97d083c81e0245d51f0734d8a6614cfe3db1315b660a8c3"} Oct 29 23:49:38.871491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.871358 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa1ab7b0fa9476ed97d083c81e0245d51f0734d8a6614cfe3db1315b660a8c3" Oct 29 23:49:38.872839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.872812 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" event={"ID":"bedc11e4-3f71-4390-a308-564524688cdd","Type":"ContainerDied","Data":"1205ebd05bf42ab4a011b73fe46cc2ae6dfdfead621b0009c2ea1cbd4a430acd"} Oct 29 23:49:38.872839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.872836 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55" Oct 29 23:49:38.873060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:38.872852 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1205ebd05bf42ab4a011b73fe46cc2ae6dfdfead621b0009c2ea1cbd4a430acd" Oct 29 23:49:44.111180 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.111144 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq"] Oct 29 23:49:44.127401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.127366 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dec3212-2v7rq"] Oct 29 23:49:44.258195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.258156 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x"] Oct 29 23:49:44.258823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.258807 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1077baad-3485-4973-951f-50e8ff9d5c0e" containerName="revoke-keys" Oct 29 23:49:44.258886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.258825 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1077baad-3485-4973-951f-50e8ff9d5c0e" containerName="revoke-keys" Oct 29 23:49:44.258886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.258846 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bedc11e4-3f71-4390-a308-564524688cdd" containerName="revoke-keys" Oct 29 23:49:44.258886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.258852 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedc11e4-3f71-4390-a308-564524688cdd" containerName="revoke-keys" Oct 29 23:49:44.258988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.258968 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1077baad-3485-4973-951f-50e8ff9d5c0e" containerName="revoke-keys" Oct 29 23:49:44.258988 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.258981 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedc11e4-3f71-4390-a308-564524688cdd" containerName="revoke-keys" Oct 29 23:49:44.263534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.263518 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.288203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.288170 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:49:44.288337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.288291 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:49:44.310893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.310862 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x"] Oct 29 23:49:44.331004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.330963 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03d5a19-rmh5x\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.331159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.331107 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03d5a19-rmh5x\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.331445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.331425 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9cmk\" (UniqueName: \"kubernetes.io/projected/64092c48-8906-4330-ae64-9e3b79faba70-kube-api-access-z9cmk\") pod \"maas-api-revoke-sub-f03d5a19-rmh5x\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.433289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.433215 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03d5a19-rmh5x\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.433473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.433452 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9cmk\" (UniqueName: \"kubernetes.io/projected/64092c48-8906-4330-ae64-9e3b79faba70-kube-api-access-z9cmk\") pod \"maas-api-revoke-sub-f03d5a19-rmh5x\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.433518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.433488 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03d5a19-rmh5x\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.434204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.434174 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03d5a19-rmh5x\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.435590 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.435559 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03d5a19-rmh5x\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.465642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.465614 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9cmk\" (UniqueName: \"kubernetes.io/projected/64092c48-8906-4330-ae64-9e3b79faba70-kube-api-access-z9cmk\") pod \"maas-api-revoke-sub-f03d5a19-rmh5x\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.578038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.578013 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:44.896924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.896899 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x"] Oct 29 23:49:44.919222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:44.919192 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" event={"ID":"64092c48-8906-4330-ae64-9e3b79faba70","Type":"ContainerStarted","Data":"e7e18cc274a9c8b77e1c148f170295b13f867c3218268598ed87d77912f5dbf6"} Oct 29 23:49:45.013846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:45.013814 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a" path="/var/lib/kubelet/pods/6cebdb8b-1c50-42bf-a3f6-79a1f5e0eb2a/volumes" Oct 29 23:49:45.931204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:45.931175 5736 generic.go:356] "Generic (PLEG): container finished" podID="64092c48-8906-4330-ae64-9e3b79faba70" containerID="11979fb2ea105e308668aef91adb14520f54a21d894726eb391a99164378c9eb" exitCode=0 Oct 29 23:49:45.931523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:45.931254 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" event={"ID":"64092c48-8906-4330-ae64-9e3b79faba70","Type":"ContainerDied","Data":"11979fb2ea105e308668aef91adb14520f54a21d894726eb391a99164378c9eb"} Oct 29 23:49:47.093338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.093314 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:47.174508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.174460 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-openshift-ca\") pod \"64092c48-8906-4330-ae64-9e3b79faba70\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " Oct 29 23:49:47.175192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.175008 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-xks-ca\") pod \"64092c48-8906-4330-ae64-9e3b79faba70\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " Oct 29 23:49:47.175192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.175154 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-openshift-ca" pod "64092c48-8906-4330-ae64-9e3b79faba70" (UID: "64092c48-8906-4330-ae64-9e3b79faba70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:49:47.175397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.175231 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64092c48-8906-4330-ae64-9e3b79faba70-kube-api-access-z9cmk\" (UniqueName: \"kubernetes.io/projected/64092c48-8906-4330-ae64-9e3b79faba70-kube-api-access-z9cmk\") pod \"64092c48-8906-4330-ae64-9e3b79faba70\" (UID: \"64092c48-8906-4330-ae64-9e3b79faba70\") " Oct 29 23:49:47.175936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.175903 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:47.178224 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.178188 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64092c48-8906-4330-ae64-9e3b79faba70-kube-api-access-z9cmk" pod "64092c48-8906-4330-ae64-9e3b79faba70" (UID: "64092c48-8906-4330-ae64-9e3b79faba70"). InnerVolumeSpecName "kube-api-access-z9cmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:49:47.178564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.178515 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-xks-ca" pod "64092c48-8906-4330-ae64-9e3b79faba70" (UID: "64092c48-8906-4330-ae64-9e3b79faba70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:49:47.276802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.276774 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64092c48-8906-4330-ae64-9e3b79faba70-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:47.276802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.276801 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9cmk\" (UniqueName: \"kubernetes.io/projected/64092c48-8906-4330-ae64-9e3b79faba70-kube-api-access-z9cmk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:49:47.948203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.948169 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" Oct 29 23:49:47.948376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.948199 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x" event={"ID":"64092c48-8906-4330-ae64-9e3b79faba70","Type":"ContainerDied","Data":"e7e18cc274a9c8b77e1c148f170295b13f867c3218268598ed87d77912f5dbf6"} Oct 29 23:49:47.948376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:47.948253 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7e18cc274a9c8b77e1c148f170295b13f867c3218268598ed87d77912f5dbf6" Oct 29 23:49:56.823555 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:49:56.823512 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:49:58.756043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.756010 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh"] Oct 29 23:49:58.756744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.756723 5736 cpu_manager.go:414] "containerMap: removing container" podUID="64092c48-8906-4330-ae64-9e3b79faba70" containerName="revoke-keys" Oct 29 23:49:58.756812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.756748 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64092c48-8906-4330-ae64-9e3b79faba70" containerName="revoke-keys" Oct 29 23:49:58.756916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.756904 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="64092c48-8906-4330-ae64-9e3b79faba70" containerName="revoke-keys" Oct 29 23:49:58.760771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.760746 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:58.776895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.776874 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:49:58.788627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.788602 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:49:58.792146 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.792122 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh"] Oct 29 23:49:58.851661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.851630 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381427e0-gwrlh\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:58.851827 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.851797 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381427e0-gwrlh\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:58.851900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.851849 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lzrq\" (UniqueName: \"kubernetes.io/projected/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-kube-api-access-6lzrq\") pod \"maas-api-revoke-sub-381427e0-gwrlh\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:58.953429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.953391 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381427e0-gwrlh\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:58.953598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.953498 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381427e0-gwrlh\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:58.953598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.953524 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lzrq\" (UniqueName: \"kubernetes.io/projected/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-kube-api-access-6lzrq\") pod \"maas-api-revoke-sub-381427e0-gwrlh\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:58.954225 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.954201 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381427e0-gwrlh\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:58.955920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.955893 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381427e0-gwrlh\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:58.976978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:58.976953 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lzrq\" (UniqueName: \"kubernetes.io/projected/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-kube-api-access-6lzrq\") pod \"maas-api-revoke-sub-381427e0-gwrlh\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:59.075729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:59.075659 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:49:59.384764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:49:59.384733 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh"] Oct 29 23:49:59.387390 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:49:59.387355 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8075cc6b_442f_4d04_bd08_cd5d2e0029f8.slice/crio-024a2ae23e3032a47fc5efb905b7f046c423cc16233f2d13aa1de93e3aaeb1df WatchSource:0}: Error finding container 024a2ae23e3032a47fc5efb905b7f046c423cc16233f2d13aa1de93e3aaeb1df: Status 404 returned error can't find the container with id 024a2ae23e3032a47fc5efb905b7f046c423cc16233f2d13aa1de93e3aaeb1df Oct 29 23:50:00.073816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:00.073773 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" event={"ID":"8075cc6b-442f-4d04-bd08-cd5d2e0029f8","Type":"ContainerStarted","Data":"5f659e85217e5f6793be2aab436f5f1481bb796a63fcbaa6e443f47e970130bc"} Oct 29 23:50:00.074223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:00.073874 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" event={"ID":"8075cc6b-442f-4d04-bd08-cd5d2e0029f8","Type":"ContainerStarted","Data":"024a2ae23e3032a47fc5efb905b7f046c423cc16233f2d13aa1de93e3aaeb1df"} Oct 29 23:50:00.117674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:00.117622 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" podStartSLOduration=2.117608144 podStartE2EDuration="2.117608144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:50:00.114850045 +0000 UTC m=+3997.230769652" watchObservedRunningTime="2026-10-29 23:50:00.117608144 +0000 UTC m=+3997.233527770" Oct 29 23:50:01.083232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:01.083192 5736 generic.go:356] "Generic (PLEG): container finished" podID="8075cc6b-442f-4d04-bd08-cd5d2e0029f8" containerID="5f659e85217e5f6793be2aab436f5f1481bb796a63fcbaa6e443f47e970130bc" exitCode=0 Oct 29 23:50:01.083671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:01.083277 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" event={"ID":"8075cc6b-442f-4d04-bd08-cd5d2e0029f8","Type":"ContainerDied","Data":"5f659e85217e5f6793be2aab436f5f1481bb796a63fcbaa6e443f47e970130bc"} Oct 29 23:50:02.248800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.248770 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:50:02.306552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.306511 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-openshift-ca\") pod \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " Oct 29 23:50:02.306767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.306610 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-kube-api-access-6lzrq\" (UniqueName: \"kubernetes.io/projected/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-kube-api-access-6lzrq\") pod \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " Oct 29 23:50:02.306851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.306774 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-xks-ca\") pod \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\" (UID: \"8075cc6b-442f-4d04-bd08-cd5d2e0029f8\") " Oct 29 23:50:02.307035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.307003 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-openshift-ca" pod "8075cc6b-442f-4d04-bd08-cd5d2e0029f8" (UID: "8075cc6b-442f-4d04-bd08-cd5d2e0029f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:02.307378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.307358 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:02.308957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.308926 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-xks-ca" pod "8075cc6b-442f-4d04-bd08-cd5d2e0029f8" (UID: "8075cc6b-442f-4d04-bd08-cd5d2e0029f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:02.309457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.309432 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-kube-api-access-6lzrq" pod "8075cc6b-442f-4d04-bd08-cd5d2e0029f8" (UID: "8075cc6b-442f-4d04-bd08-cd5d2e0029f8"). InnerVolumeSpecName "kube-api-access-6lzrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:02.408320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.408228 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:02.408320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:02.408268 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lzrq\" (UniqueName: \"kubernetes.io/projected/8075cc6b-442f-4d04-bd08-cd5d2e0029f8-kube-api-access-6lzrq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:03.102398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:03.102368 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" Oct 29 23:50:03.102574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:03.102399 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh" event={"ID":"8075cc6b-442f-4d04-bd08-cd5d2e0029f8","Type":"ContainerDied","Data":"024a2ae23e3032a47fc5efb905b7f046c423cc16233f2d13aa1de93e3aaeb1df"} Oct 29 23:50:03.102574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:03.102449 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="024a2ae23e3032a47fc5efb905b7f046c423cc16233f2d13aa1de93e3aaeb1df" Oct 29 23:50:06.919179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:06.919143 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c"] Oct 29 23:50:06.929277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:06.929248 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ed8218-r4z5c"] Oct 29 23:50:07.006424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:07.006384 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db88598-c68d-423e-ac5b-4cb29ea4d52e" path="/var/lib/kubelet/pods/9db88598-c68d-423e-ac5b-4cb29ea4d52e/volumes" Oct 29 23:50:14.358773 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:14.358736 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k"] Oct 29 23:50:14.359545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:14.359506 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" containerID="cri-o://4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c" gracePeriod=30 Oct 29 23:50:15.798379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:15.798333 5736 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:50:15.798846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:15.798392 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:50:19.390756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.390721 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk"] Oct 29 23:50:19.706456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.706375 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg"] Oct 29 23:50:19.707048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.707028 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8075cc6b-442f-4d04-bd08-cd5d2e0029f8" containerName="revoke-keys" Oct 29 23:50:19.707119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.707049 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8075cc6b-442f-4d04-bd08-cd5d2e0029f8" containerName="revoke-keys" Oct 29 23:50:19.707222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.707211 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8075cc6b-442f-4d04-bd08-cd5d2e0029f8" containerName="revoke-keys" Oct 29 23:50:19.710883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.710861 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:19.732915 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.732891 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg"] Oct 29 23:50:19.733526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.733501 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:50:19.733640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.733553 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:50:19.828676 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.828633 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cx8n\" (UniqueName: \"kubernetes.io/projected/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-kube-api-access-9cx8n\") pod \"maas-api-revoke-keys-e2e-inf-a-750972-ltrgg\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:19.828878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.828702 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-750972-ltrgg\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:19.828878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.828850 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-750972-ltrgg\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:19.931426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.931117 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cx8n\" (UniqueName: \"kubernetes.io/projected/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-kube-api-access-9cx8n\") pod \"maas-api-revoke-keys-e2e-inf-a-750972-ltrgg\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:19.931426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.931197 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-750972-ltrgg\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:19.931426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.931268 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-750972-ltrgg\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:19.932122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.932059 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-750972-ltrgg\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:19.935157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.935129 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-750972-ltrgg\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:19.958293 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:19.958207 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cx8n\" (UniqueName: \"kubernetes.io/projected/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-kube-api-access-9cx8n\") pod \"maas-api-revoke-keys-e2e-inf-a-750972-ltrgg\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:20.025116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.025076 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:20.078426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.078398 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 23:50:20.134594 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.134540 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-data\") pod \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " Oct 29 23:50:20.134785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.134636 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-podinfo\") pod \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " Oct 29 23:50:20.134785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.134755 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-envoy\") pod \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " Oct 29 23:50:20.134900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.134801 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-token\") pod \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " Oct 29 23:50:20.134955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.134907 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-certs\") pod \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " Oct 29 23:50:20.134955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.134947 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istiod-ca-cert\") pod \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " Oct 29 23:50:20.135121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.134971 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-data" pod "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" (UID: "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:20.135121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.135011 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-socket\") pod \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " Oct 29 23:50:20.135230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.135131 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-kube-api-access-f66gw\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-kube-api-access-f66gw\") pod \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " Oct 29 23:50:20.135230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.135161 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-credential-socket\") pod \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\" (UID: \"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1\") " Oct 29 23:50:20.135985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.135955 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.135985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.135974 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istiod-ca-cert" pod "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" (UID: "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:20.136399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.136351 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-certs" pod "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" (UID: "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:20.136502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.136417 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-socket" pod "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" (UID: "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:20.136808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.136785 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-credential-socket" pod "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" (UID: "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:20.137778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.137748 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-token" pod "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" (UID: "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:20.138206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.138184 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-envoy" pod "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" (UID: "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:20.138350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.138333 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-podinfo" pod "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" (UID: "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:50:20.138824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.138799 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-kube-api-access-f66gw" pod "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" (UID: "ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1"). InnerVolumeSpecName "kube-api-access-f66gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:20.237147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.237113 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66gw\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-kube-api-access-f66gw\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.237147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.237146 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.237359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.237156 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.237359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.237165 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.237359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.237180 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.237359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.237190 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.237359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.237198 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.237359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.237206 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:20.307359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.307325 5736 generic.go:356] "Generic (PLEG): container finished" podID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerID="4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c" exitCode=0 Oct 29 23:50:20.307548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.307377 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" Oct 29 23:50:20.307548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.307378 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" event={"ID":"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1","Type":"ContainerDied","Data":"4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c"} Oct 29 23:50:20.307548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.307506 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k" event={"ID":"ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1","Type":"ContainerDied","Data":"f3aff2d8a6ee9165fad799836958d9208258f40d0c6440805e9fc1f8c04695fe"} Oct 29 23:50:20.307548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.307533 5736 scope.go:122] "RemoveContainer" containerID="4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c" Oct 29 23:50:20.319388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.319368 5736 scope.go:122] "RemoveContainer" containerID="4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c" Oct 29 23:50:20.319668 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:20.319644 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c\": container with ID starting with 4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c not found: ID does not exist" containerID="4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c" Oct 29 23:50:20.319777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.319675 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c"} err="failed to get container status \"4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c\": rpc error: code = NotFound desc = could not find container \"4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c\": container with ID starting with 4f9ae8b0b3f93962a24ceffc975c1369349630cebe55f0f2f8a82b936d27bb3c not found: ID does not exist" Oct 29 23:50:20.342569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.342541 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg"] Oct 29 23:50:20.344327 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:50:20.344304 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a86dd4_3e11_43cf_b45f_72ca47b79e66.slice/crio-7b98f8b6416eb71a066a461550e941f6e33e7998ac9540742521982b80b2d707 WatchSource:0}: Error finding container 7b98f8b6416eb71a066a461550e941f6e33e7998ac9540742521982b80b2d707: Status 404 returned error can't find the container with id 7b98f8b6416eb71a066a461550e941f6e33e7998ac9540742521982b80b2d707 Oct 29 23:50:20.364831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.364790 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k"] Oct 29 23:50:20.376617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:20.373201 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-kq65k"] Oct 29 23:50:21.005185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:21.005145 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" path="/var/lib/kubelet/pods/ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1/volumes" Oct 29 23:50:21.317383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:21.317300 5736 generic.go:356] "Generic (PLEG): container finished" podID="e2a86dd4-3e11-43cf-b45f-72ca47b79e66" containerID="ea2d9879c8b7370e7f83d5bb803d788ce8a4bc87ae33500240553b99e48338e6" exitCode=0 Oct 29 23:50:21.317541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:21.317376 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" event={"ID":"e2a86dd4-3e11-43cf-b45f-72ca47b79e66","Type":"ContainerDied","Data":"ea2d9879c8b7370e7f83d5bb803d788ce8a4bc87ae33500240553b99e48338e6"} Oct 29 23:50:21.317541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:21.317428 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" event={"ID":"e2a86dd4-3e11-43cf-b45f-72ca47b79e66","Type":"ContainerStarted","Data":"7b98f8b6416eb71a066a461550e941f6e33e7998ac9540742521982b80b2d707"} Oct 29 23:50:22.556006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.555970 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:22.669651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.669207 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-kube-api-access-9cx8n\" (UniqueName: \"kubernetes.io/projected/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-kube-api-access-9cx8n\") pod \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " Oct 29 23:50:22.669651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.669370 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-openshift-ca\") pod \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " Oct 29 23:50:22.669651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.669569 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-xks-ca\") pod \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\" (UID: \"e2a86dd4-3e11-43cf-b45f-72ca47b79e66\") " Oct 29 23:50:22.678871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.678838 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-openshift-ca" pod "e2a86dd4-3e11-43cf-b45f-72ca47b79e66" (UID: "e2a86dd4-3e11-43cf-b45f-72ca47b79e66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:22.683062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.683027 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-xks-ca" pod "e2a86dd4-3e11-43cf-b45f-72ca47b79e66" (UID: "e2a86dd4-3e11-43cf-b45f-72ca47b79e66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:22.722723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.722677 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-kube-api-access-9cx8n" pod "e2a86dd4-3e11-43cf-b45f-72ca47b79e66" (UID: "e2a86dd4-3e11-43cf-b45f-72ca47b79e66"). InnerVolumeSpecName "kube-api-access-9cx8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:22.771423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.771387 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:22.771423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.771420 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:22.771423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:22.771430 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cx8n\" (UniqueName: \"kubernetes.io/projected/e2a86dd4-3e11-43cf-b45f-72ca47b79e66-kube-api-access-9cx8n\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:23.338643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:23.338557 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" Oct 29 23:50:23.338643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:23.338574 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg" event={"ID":"e2a86dd4-3e11-43cf-b45f-72ca47b79e66","Type":"ContainerDied","Data":"7b98f8b6416eb71a066a461550e941f6e33e7998ac9540742521982b80b2d707"} Oct 29 23:50:23.338837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:23.338660 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b98f8b6416eb71a066a461550e941f6e33e7998ac9540742521982b80b2d707" Oct 29 23:50:23.348039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:23.348013 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs"] Oct 29 23:50:23.359998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:23.359973 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2591f6-wwqrs"] Oct 29 23:50:25.004633 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:25.004595 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279f8c23-d226-4604-bdca-bf0fe8c1164c" path="/var/lib/kubelet/pods/279f8c23-d226-4604-bdca-bf0fe8c1164c/volumes" Oct 29 23:50:26.836953 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:26.836908 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:50:27.718648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:27.718608 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk"] Oct 29 23:50:27.722346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:27.722319 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e57216-5bhxk"] Oct 29 23:50:29.004209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:29.004172 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2d7af5-e65f-47c1-9323-9851b1721066" path="/var/lib/kubelet/pods/af2d7af5-e65f-47c1-9323-9851b1721066/volumes" Oct 29 23:50:31.287906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:31.287870 5736 scope.go:122] "RemoveContainer" containerID="88b7d712a3c624a91dd08d9f90939f5ad9315373aa21add0d67bf11f650973e5" Oct 29 23:50:31.299105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:31.299081 5736 scope.go:122] "RemoveContainer" containerID="0597373c2cdad28c9b8abf8e90f29b0e6f94e718a6e9ddbb40ee6a836706a5c3" Oct 29 23:50:31.308684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:31.308658 5736 scope.go:122] "RemoveContainer" containerID="a0122019a4cbac13ce9f1cbe60b34b4d2931aaebbae6d27c74ac9920c4c25395" Oct 29 23:50:32.106641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.106605 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg"] Oct 29 23:50:32.106907 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.106874 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" containerID="cri-o://2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9" gracePeriod=30 Oct 29 23:50:32.285631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.280176 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn"] Oct 29 23:50:32.315607 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.315548 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" Oct 29 23:50:32.316142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.316115 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" Oct 29 23:50:32.316264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.316250 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e2a86dd4-3e11-43cf-b45f-72ca47b79e66" containerName="revoke-keys" Oct 29 23:50:32.316347 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.316335 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a86dd4-3e11-43cf-b45f-72ca47b79e66" containerName="revoke-keys" Oct 29 23:50:32.316745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.316723 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0c30d7-737c-4e13-ad54-4b7f7cb8a7f1" containerName="istio-proxy" Oct 29 23:50:32.316873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.316861 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a86dd4-3e11-43cf-b45f-72ca47b79e66" containerName="revoke-keys" Oct 29 23:50:32.330513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.330476 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn"] Oct 29 23:50:32.330941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.330918 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.348234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.348212 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:50:32.348494 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.348476 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:50:32.380557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.380460 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-269a18c6-kwwkn\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.380732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.380654 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhkk2\" (UniqueName: \"kubernetes.io/projected/351f64d9-561d-49be-a84b-14b91010bcec-kube-api-access-qhkk2\") pod \"maas-api-revoke-sub-269a18c6-kwwkn\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.380732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.380691 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-269a18c6-kwwkn\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.482037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.481997 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-269a18c6-kwwkn\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.482243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.482085 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhkk2\" (UniqueName: \"kubernetes.io/projected/351f64d9-561d-49be-a84b-14b91010bcec-kube-api-access-qhkk2\") pod \"maas-api-revoke-sub-269a18c6-kwwkn\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.482243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.482114 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-269a18c6-kwwkn\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.482826 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.482800 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-269a18c6-kwwkn\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.484571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.484539 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-269a18c6-kwwkn\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.505122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.505100 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhkk2\" (UniqueName: \"kubernetes.io/projected/351f64d9-561d-49be-a84b-14b91010bcec-kube-api-access-qhkk2\") pod \"maas-api-revoke-sub-269a18c6-kwwkn\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.657332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.657263 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:32.705660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.705623 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95"] Oct 29 23:50:32.710153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.710128 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.736067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.735774 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95"] Oct 29 23:50:32.785531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.785500 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.785745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.785719 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.785866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.785849 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6dk\" (UniqueName: \"kubernetes.io/projected/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-kube-api-access-hk6dk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.886853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.886818 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6dk\" (UniqueName: \"kubernetes.io/projected/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-kube-api-access-hk6dk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.886981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.886948 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.887114 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.887098 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.887628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.887573 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.889506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.889485 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.910611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.910545 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6dk\" (UniqueName: \"kubernetes.io/projected/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-kube-api-access-hk6dk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:32.967969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:32.967939 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn"] Oct 29 23:50:32.970530 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:50:32.970493 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351f64d9_561d_49be_a84b_14b91010bcec.slice/crio-7ddadb7118ee0bebebadace7a420ac2f8dc6dc51f26369ee40426d48127d059d WatchSource:0}: Error finding container 7ddadb7118ee0bebebadace7a420ac2f8dc6dc51f26369ee40426d48127d059d: Status 404 returned error can't find the container with id 7ddadb7118ee0bebebadace7a420ac2f8dc6dc51f26369ee40426d48127d059d Oct 29 23:50:33.039484 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:33.039457 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:33.346209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:33.346181 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95"] Oct 29 23:50:33.348495 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:50:33.348467 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a19c33_3b10_4bce_ba8f_3fa01ab9a5c7.slice/crio-41880fd3424060968c0d218e3d860c4495feeb6302ec0c5116b475186022d9e4 WatchSource:0}: Error finding container 41880fd3424060968c0d218e3d860c4495feeb6302ec0c5116b475186022d9e4: Status 404 returned error can't find the container with id 41880fd3424060968c0d218e3d860c4495feeb6302ec0c5116b475186022d9e4 Oct 29 23:50:33.482554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:33.482512 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" event={"ID":"351f64d9-561d-49be-a84b-14b91010bcec","Type":"ContainerStarted","Data":"196bdb878955a2d07730e1020ae66ec89f7e703b9be288f378ca6cb2fc6d58b5"} Oct 29 23:50:33.482740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:33.482655 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" event={"ID":"351f64d9-561d-49be-a84b-14b91010bcec","Type":"ContainerStarted","Data":"7ddadb7118ee0bebebadace7a420ac2f8dc6dc51f26369ee40426d48127d059d"} Oct 29 23:50:33.485330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:33.485294 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" event={"ID":"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7","Type":"ContainerStarted","Data":"41880fd3424060968c0d218e3d860c4495feeb6302ec0c5116b475186022d9e4"} Oct 29 23:50:33.530233 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:33.530184 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" podStartSLOduration=1.530167289 podStartE2EDuration="1.530167289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:50:33.525792777 +0000 UTC m=+4030.641712383" watchObservedRunningTime="2026-10-29 23:50:33.530167289 +0000 UTC m=+4030.646086895" Oct 29 23:50:34.440951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.440913 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" podUID="f7dc7648-76f9-4d85-81ea-ece60b3f76ae" containerName="main" containerID="cri-o://00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4" gracePeriod=45 Oct 29 23:50:34.450785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.450755 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" podUID="f7dc7648-76f9-4d85-81ea-ece60b3f76ae" containerName="main" probeResult="failure" output="Get \"https://10.134.1.130:8000/ready\": dial tcp 10.134.1.130:8000: connect: connection refused" Oct 29 23:50:34.494450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.494422 5736 generic.go:356] "Generic (PLEG): container finished" podID="45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7" containerID="f6299d31ede4ad429cbc1fbf51ce5701ea1a5869616e7547eee17a37b6369d78" exitCode=0 Oct 29 23:50:34.494570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.494503 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" event={"ID":"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7","Type":"ContainerDied","Data":"f6299d31ede4ad429cbc1fbf51ce5701ea1a5869616e7547eee17a37b6369d78"} Oct 29 23:50:34.496310 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.496290 5736 generic.go:356] "Generic (PLEG): container finished" podID="351f64d9-561d-49be-a84b-14b91010bcec" containerID="196bdb878955a2d07730e1020ae66ec89f7e703b9be288f378ca6cb2fc6d58b5" exitCode=0 Oct 29 23:50:34.496418 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.496370 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" event={"ID":"351f64d9-561d-49be-a84b-14b91010bcec","Type":"ContainerDied","Data":"196bdb878955a2d07730e1020ae66ec89f7e703b9be288f378ca6cb2fc6d58b5"} Oct 29 23:50:34.714529 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.714503 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:50:34.838480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.838403 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-dshm\") pod \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " Oct 29 23:50:34.838480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.838455 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-model-cache\") pod \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " Oct 29 23:50:34.838716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.838502 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-home\") pod \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " Oct 29 23:50:34.838784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.838754 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-model-cache" pod "f7dc7648-76f9-4d85-81ea-ece60b3f76ae" (UID: "f7dc7648-76f9-4d85-81ea-ece60b3f76ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:34.838843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.838781 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tmp-dir\") pod \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " Oct 29 23:50:34.838843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.838820 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tls-certs\") pod \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " Oct 29 23:50:34.838961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.838850 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-home" pod "f7dc7648-76f9-4d85-81ea-ece60b3f76ae" (UID: "f7dc7648-76f9-4d85-81ea-ece60b3f76ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:34.838961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.838883 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-kube-api-access-hgm67\" (UniqueName: \"kubernetes.io/projected/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-kube-api-access-hgm67\") pod \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\" (UID: \"f7dc7648-76f9-4d85-81ea-ece60b3f76ae\") " Oct 29 23:50:34.839079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.839012 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tmp-dir" pod "f7dc7648-76f9-4d85-81ea-ece60b3f76ae" (UID: "f7dc7648-76f9-4d85-81ea-ece60b3f76ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:34.839463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.839437 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:34.839463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.839465 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:34.839678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.839477 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:34.840917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.840887 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tls-certs" pod "f7dc7648-76f9-4d85-81ea-ece60b3f76ae" (UID: "f7dc7648-76f9-4d85-81ea-ece60b3f76ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:34.841020 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.840893 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-dshm" pod "f7dc7648-76f9-4d85-81ea-ece60b3f76ae" (UID: "f7dc7648-76f9-4d85-81ea-ece60b3f76ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:34.841070 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.841037 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-kube-api-access-hgm67" pod "f7dc7648-76f9-4d85-81ea-ece60b3f76ae" (UID: "f7dc7648-76f9-4d85-81ea-ece60b3f76ae"). InnerVolumeSpecName "kube-api-access-hgm67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:34.941090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.941063 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:34.941090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.941089 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgm67\" (UniqueName: \"kubernetes.io/projected/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-kube-api-access-hgm67\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:34.941256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:34.941099 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7648-76f9-4d85-81ea-ece60b3f76ae-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.432625 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.432565 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt"] Oct 29 23:50:35.432854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.432827 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" containerID="cri-o://dbe537a4c8b7b3f860000466c566e5bed205f6ce5b6f83b9eb890d1b50f499fb" gracePeriod=30 Oct 29 23:50:35.505385 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.505351 5736 generic.go:356] "Generic (PLEG): container finished" podID="f7dc7648-76f9-4d85-81ea-ece60b3f76ae" containerID="00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4" exitCode=0 Oct 29 23:50:35.505873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.505390 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" Oct 29 23:50:35.505873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.505429 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" event={"ID":"f7dc7648-76f9-4d85-81ea-ece60b3f76ae","Type":"ContainerDied","Data":"00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4"} Oct 29 23:50:35.505873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.505474 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk" event={"ID":"f7dc7648-76f9-4d85-81ea-ece60b3f76ae","Type":"ContainerDied","Data":"b5e9d2751ab32aa38926d64862c049acfb8f74eaa3edc9fa6370b9ac3778e707"} Oct 29 23:50:35.505873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.505497 5736 scope.go:122] "RemoveContainer" containerID="00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4" Oct 29 23:50:35.645039 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.644613 5736 scope.go:122] "RemoveContainer" containerID="00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4" Oct 29 23:50:35.645511 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:35.645441 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4\": container with ID starting with 00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4 not found: ID does not exist" containerID="00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4" Oct 29 23:50:35.645511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.645475 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4"} err="failed to get container status \"00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4\": rpc error: code = NotFound desc = could not find container \"00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4\": container with ID starting with 00b424bc8815022c8d31b3dcfe98b86217d363c1c3eae13d4b855a7b90c1a5c4 not found: ID does not exist" Oct 29 23:50:35.653206 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.649641 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk"] Oct 29 23:50:35.654230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.654195 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d20345/test-model-d20345-kserve-6bf7495cf5-4fsbk"] Oct 29 23:50:35.828134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.828108 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:35.830618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.830563 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:35.916290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.916253 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-openshift-ca\") pod \"351f64d9-561d-49be-a84b-14b91010bcec\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " Oct 29 23:50:35.916480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.916385 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-xks-ca\") pod \"351f64d9-561d-49be-a84b-14b91010bcec\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " Oct 29 23:50:35.916480 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.916437 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-xks-ca\") pod \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " Oct 29 23:50:35.916643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.916490 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-openshift-ca\") pod \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " Oct 29 23:50:35.916643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.916544 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-kube-api-access-hk6dk\" (UniqueName: \"kubernetes.io/projected/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-kube-api-access-hk6dk\") pod \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\" (UID: \"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7\") " Oct 29 23:50:35.916643 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.916572 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351f64d9-561d-49be-a84b-14b91010bcec-kube-api-access-qhkk2\" (UniqueName: \"kubernetes.io/projected/351f64d9-561d-49be-a84b-14b91010bcec-kube-api-access-qhkk2\") pod \"351f64d9-561d-49be-a84b-14b91010bcec\" (UID: \"351f64d9-561d-49be-a84b-14b91010bcec\") " Oct 29 23:50:35.916856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.916827 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-openshift-ca" pod "351f64d9-561d-49be-a84b-14b91010bcec" (UID: "351f64d9-561d-49be-a84b-14b91010bcec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:35.917317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.917287 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:35.918198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.918170 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-openshift-ca" pod "45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7" (UID: "45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:35.920732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.920706 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-xks-ca" pod "45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7" (UID: "45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:35.920851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.920828 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-kube-api-access-hk6dk" pod "45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7" (UID: "45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7"). InnerVolumeSpecName "kube-api-access-hk6dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:35.921486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.921456 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-xks-ca" pod "351f64d9-561d-49be-a84b-14b91010bcec" (UID: "351f64d9-561d-49be-a84b-14b91010bcec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:35.922487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:35.922466 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351f64d9-561d-49be-a84b-14b91010bcec-kube-api-access-qhkk2" pod "351f64d9-561d-49be-a84b-14b91010bcec" (UID: "351f64d9-561d-49be-a84b-14b91010bcec"). InnerVolumeSpecName "kube-api-access-qhkk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:36.018953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.018917 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351f64d9-561d-49be-a84b-14b91010bcec-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:36.018953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.018948 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:36.018953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.018958 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:36.019226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.018969 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk6dk\" (UniqueName: \"kubernetes.io/projected/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7-kube-api-access-hk6dk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:36.019226 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.018980 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhkk2\" (UniqueName: \"kubernetes.io/projected/351f64d9-561d-49be-a84b-14b91010bcec-kube-api-access-qhkk2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:36.516175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.516140 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" Oct 29 23:50:36.516175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.516149 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95" event={"ID":"45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7","Type":"ContainerDied","Data":"41880fd3424060968c0d218e3d860c4495feeb6302ec0c5116b475186022d9e4"} Oct 29 23:50:36.516755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.516210 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41880fd3424060968c0d218e3d860c4495feeb6302ec0c5116b475186022d9e4" Oct 29 23:50:36.518081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.518042 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" Oct 29 23:50:36.518253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.518076 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn" event={"ID":"351f64d9-561d-49be-a84b-14b91010bcec","Type":"ContainerDied","Data":"7ddadb7118ee0bebebadace7a420ac2f8dc6dc51f26369ee40426d48127d059d"} Oct 29 23:50:36.518253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:36.518122 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ddadb7118ee0bebebadace7a420ac2f8dc6dc51f26369ee40426d48127d059d" Oct 29 23:50:37.004621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.004564 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7dc7648-76f9-4d85-81ea-ece60b3f76ae" path="/var/lib/kubelet/pods/f7dc7648-76f9-4d85-81ea-ece60b3f76ae/volumes" Oct 29 23:50:37.409522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.409497 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:50:37.503782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.503744 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z"] Oct 29 23:50:37.504309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.503959 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" podUID="64e5d601-93ef-4424-b06c-bc5ca97943e5" containerName="maas-api" containerID="cri-o://e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5" gracePeriod=30 Oct 29 23:50:37.534454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.534417 5736 generic.go:356] "Generic (PLEG): container finished" podID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerID="2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9" exitCode=0 Oct 29 23:50:37.534945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.534483 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" Oct 29 23:50:37.534945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.534497 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" event={"ID":"b002e9fa-a1b2-4a58-9468-0a6de1a35017","Type":"ContainerDied","Data":"2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9"} Oct 29 23:50:37.534945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.534570 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg" event={"ID":"b002e9fa-a1b2-4a58-9468-0a6de1a35017","Type":"ContainerDied","Data":"cf40de5621727c3573d32fb8cb776c4226c09946cbae907fcf1a613280de7c4f"} Oct 29 23:50:37.534945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.534638 5736 scope.go:122] "RemoveContainer" containerID="2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9" Oct 29 23:50:37.545823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.545758 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-token\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-token\") pod \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " Oct 29 23:50:37.556608 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.556569 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-kube-api-access-sfsxj\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-kube-api-access-sfsxj\") pod \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " Oct 29 23:50:37.557159 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.557133 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istiod-ca-cert\") pod \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " Oct 29 23:50:37.557335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.557315 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-data\") pod \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " Oct 29 23:50:37.557424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.557361 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-envoy\") pod \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " Oct 29 23:50:37.557485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.557419 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-socket\") pod \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " Oct 29 23:50:37.557485 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.557472 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-podinfo\") pod \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " Oct 29 23:50:37.557670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.557652 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-certs\") pod \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " Oct 29 23:50:37.557751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.557737 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-credential-socket\") pod \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\" (UID: \"b002e9fa-a1b2-4a58-9468-0a6de1a35017\") " Oct 29 23:50:37.559120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.559094 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-credential-socket" pod "b002e9fa-a1b2-4a58-9468-0a6de1a35017" (UID: "b002e9fa-a1b2-4a58-9468-0a6de1a35017"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:37.559434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.559413 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istiod-ca-cert" pod "b002e9fa-a1b2-4a58-9468-0a6de1a35017" (UID: "b002e9fa-a1b2-4a58-9468-0a6de1a35017"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:37.559951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.559806 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-certs" pod "b002e9fa-a1b2-4a58-9468-0a6de1a35017" (UID: "b002e9fa-a1b2-4a58-9468-0a6de1a35017"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:37.559951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.559912 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-data" pod "b002e9fa-a1b2-4a58-9468-0a6de1a35017" (UID: "b002e9fa-a1b2-4a58-9468-0a6de1a35017"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:37.561510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.561367 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-socket" pod "b002e9fa-a1b2-4a58-9468-0a6de1a35017" (UID: "b002e9fa-a1b2-4a58-9468-0a6de1a35017"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:37.569392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.569364 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-token" pod "b002e9fa-a1b2-4a58-9468-0a6de1a35017" (UID: "b002e9fa-a1b2-4a58-9468-0a6de1a35017"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:37.569981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.569931 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-kube-api-access-sfsxj" pod "b002e9fa-a1b2-4a58-9468-0a6de1a35017" (UID: "b002e9fa-a1b2-4a58-9468-0a6de1a35017"). InnerVolumeSpecName "kube-api-access-sfsxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:37.574147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.574118 5736 scope.go:122] "RemoveContainer" containerID="27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61" Oct 29 23:50:37.575530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.575371 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-envoy" pod "b002e9fa-a1b2-4a58-9468-0a6de1a35017" (UID: "b002e9fa-a1b2-4a58-9468-0a6de1a35017"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:37.575530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.575474 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-podinfo" pod "b002e9fa-a1b2-4a58-9468-0a6de1a35017" (UID: "b002e9fa-a1b2-4a58-9468-0a6de1a35017"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:50:37.599372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.599343 5736 scope.go:122] "RemoveContainer" containerID="2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9" Oct 29 23:50:37.599708 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:37.599683 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9\": container with ID starting with 2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9 not found: ID does not exist" containerID="2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9" Oct 29 23:50:37.599816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.599716 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9"} err="failed to get container status \"2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9\": rpc error: code = NotFound desc = could not find container \"2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9\": container with ID starting with 2bf814db5798615cc30ee3ad6c6bac34f61ef6195deefbea9fbad27776246dc9 not found: ID does not exist" Oct 29 23:50:37.599816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.599733 5736 scope.go:122] "RemoveContainer" containerID="27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61" Oct 29 23:50:37.600030 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:37.600010 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61\": container with ID starting with 27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61 not found: ID does not exist" containerID="27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61" Oct 29 23:50:37.600081 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.600035 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61"} err="failed to get container status \"27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61\": rpc error: code = NotFound desc = could not find container \"27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61\": container with ID starting with 27ffff4e20fc6ff2fbd3d1f19e0cf6d91ae4c3e9d01972299062c394789d9d61 not found: ID does not exist" Oct 29 23:50:37.660512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.660482 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:37.660512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.660508 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:37.660512 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.660518 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:37.660755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.660527 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:37.660755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.660537 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:37.660755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.660546 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b002e9fa-a1b2-4a58-9468-0a6de1a35017-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:37.660755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.660554 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:37.660755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.660563 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfsxj\" (UniqueName: \"kubernetes.io/projected/b002e9fa-a1b2-4a58-9468-0a6de1a35017-kube-api-access-sfsxj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:37.660755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.660571 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b002e9fa-a1b2-4a58-9468-0a6de1a35017-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:37.894220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.894152 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg"] Oct 29 23:50:37.899248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:37.899125 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-6smbg"] Oct 29 23:50:38.119175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.119147 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:50:38.271029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.270996 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-metrics-certs\") pod \"64e5d601-93ef-4424-b06c-bc5ca97943e5\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " Oct 29 23:50:38.271217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.271098 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e5d601-93ef-4424-b06c-bc5ca97943e5-kube-api-access-64p4l\" (UniqueName: \"kubernetes.io/projected/64e5d601-93ef-4424-b06c-bc5ca97943e5-kube-api-access-64p4l\") pod \"64e5d601-93ef-4424-b06c-bc5ca97943e5\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " Oct 29 23:50:38.271217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.271176 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls\") pod \"64e5d601-93ef-4424-b06c-bc5ca97943e5\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " Oct 29 23:50:38.271357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.271285 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e5d601-93ef-4424-b06c-bc5ca97943e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5d601-93ef-4424-b06c-bc5ca97943e5-openshift-service-ca\") pod \"64e5d601-93ef-4424-b06c-bc5ca97943e5\" (UID: \"64e5d601-93ef-4424-b06c-bc5ca97943e5\") " Oct 29 23:50:38.271901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.271720 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e5d601-93ef-4424-b06c-bc5ca97943e5-openshift-service-ca" pod "64e5d601-93ef-4424-b06c-bc5ca97943e5" (UID: "64e5d601-93ef-4424-b06c-bc5ca97943e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:38.271901 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.271879 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5d601-93ef-4424-b06c-bc5ca97943e5-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:38.273117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.273090 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-metrics-certs" pod "64e5d601-93ef-4424-b06c-bc5ca97943e5" (UID: "64e5d601-93ef-4424-b06c-bc5ca97943e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:38.273278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.273255 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e5d601-93ef-4424-b06c-bc5ca97943e5-kube-api-access-64p4l" pod "64e5d601-93ef-4424-b06c-bc5ca97943e5" (UID: "64e5d601-93ef-4424-b06c-bc5ca97943e5"). InnerVolumeSpecName "kube-api-access-64p4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:38.273350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.273320 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls" pod "64e5d601-93ef-4424-b06c-bc5ca97943e5" (UID: "64e5d601-93ef-4424-b06c-bc5ca97943e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:38.373154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.373085 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:38.373154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.373115 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64p4l\" (UniqueName: \"kubernetes.io/projected/64e5d601-93ef-4424-b06c-bc5ca97943e5-kube-api-access-64p4l\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:38.373154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.373125 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e5d601-93ef-4424-b06c-bc5ca97943e5-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:38.492760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.492726 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg"] Oct 29 23:50:38.500313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.500279 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-750972-ltrgg"] Oct 29 23:50:38.568008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.567967 5736 generic.go:356] "Generic (PLEG): container finished" podID="64e5d601-93ef-4424-b06c-bc5ca97943e5" containerID="e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5" exitCode=0 Oct 29 23:50:38.568454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.568013 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" Oct 29 23:50:38.568454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.568252 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" event={"ID":"64e5d601-93ef-4424-b06c-bc5ca97943e5","Type":"ContainerDied","Data":"e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5"} Oct 29 23:50:38.568454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.568297 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z" event={"ID":"64e5d601-93ef-4424-b06c-bc5ca97943e5","Type":"ContainerDied","Data":"57be42fd5853065685450efea418ce7a0a4721938048ec5c68fd3baa0e28657c"} Oct 29 23:50:38.568454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.568328 5736 scope.go:122] "RemoveContainer" containerID="e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5" Oct 29 23:50:38.583723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.583700 5736 scope.go:122] "RemoveContainer" containerID="e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5" Oct 29 23:50:38.584031 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:38.584010 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5\": container with ID starting with e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5 not found: ID does not exist" containerID="e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5" Oct 29 23:50:38.584088 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.584039 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5"} err="failed to get container status \"e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5\": rpc error: code = NotFound desc = could not find container \"e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5\": container with ID starting with e9a26a964a0c9f2ee24fc0e81a90cd78a5b1de4003ae474485d9e7870a62dcf5 not found: ID does not exist" Oct 29 23:50:38.620795 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.620757 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z"] Oct 29 23:50:38.625715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:38.625665 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-750972-9bf947464-z9l7z"] Oct 29 23:50:39.005963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:39.005927 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e5d601-93ef-4424-b06c-bc5ca97943e5" path="/var/lib/kubelet/pods/64e5d601-93ef-4424-b06c-bc5ca97943e5/volumes" Oct 29 23:50:39.006432 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:39.006417 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" path="/var/lib/kubelet/pods/b002e9fa-a1b2-4a58-9468-0a6de1a35017/volumes" Oct 29 23:50:39.006875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:39.006859 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a86dd4-3e11-43cf-b45f-72ca47b79e66" path="/var/lib/kubelet/pods/e2a86dd4-3e11-43cf-b45f-72ca47b79e66/volumes" Oct 29 23:50:40.592398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.592367 5736 generic.go:356] "Generic (PLEG): container finished" podID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerID="dbe537a4c8b7b3f860000466c566e5bed205f6ce5b6f83b9eb890d1b50f499fb" exitCode=0 Oct 29 23:50:40.592876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.592450 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" event={"ID":"ef78a262-e18e-4ec5-ae50-226bc71789a1","Type":"ContainerDied","Data":"dbe537a4c8b7b3f860000466c566e5bed205f6ce5b6f83b9eb890d1b50f499fb"} Oct 29 23:50:40.592876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.592520 5736 scope.go:122] "RemoveContainer" containerID="fadbc2901818a85a4df9a69d5944cec3b8612a17b5678e5927ba2dd467fcaae4" Oct 29 23:50:40.720111 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.720047 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 23:50:40.904349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904312 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-certs\") pod \"ef78a262-e18e-4ec5-ae50-226bc71789a1\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " Oct 29 23:50:40.904349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904348 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-envoy\") pod \"ef78a262-e18e-4ec5-ae50-226bc71789a1\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " Oct 29 23:50:40.904629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904413 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-socket\") pod \"ef78a262-e18e-4ec5-ae50-226bc71789a1\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " Oct 29 23:50:40.904629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904451 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-token\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-token\") pod \"ef78a262-e18e-4ec5-ae50-226bc71789a1\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " Oct 29 23:50:40.904629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904486 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef78a262-e18e-4ec5-ae50-226bc71789a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78a262-e18e-4ec5-ae50-226bc71789a1-istiod-ca-cert\") pod \"ef78a262-e18e-4ec5-ae50-226bc71789a1\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " Oct 29 23:50:40.904629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904553 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-podinfo\") pod \"ef78a262-e18e-4ec5-ae50-226bc71789a1\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " Oct 29 23:50:40.904629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904571 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-data\") pod \"ef78a262-e18e-4ec5-ae50-226bc71789a1\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " Oct 29 23:50:40.904943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904662 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-certs" pod "ef78a262-e18e-4ec5-ae50-226bc71789a1" (UID: "ef78a262-e18e-4ec5-ae50-226bc71789a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:40.904943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904686 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-kube-api-access-tnnfv\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-kube-api-access-tnnfv\") pod \"ef78a262-e18e-4ec5-ae50-226bc71789a1\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " Oct 29 23:50:40.904943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904785 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-credential-socket\") pod \"ef78a262-e18e-4ec5-ae50-226bc71789a1\" (UID: \"ef78a262-e18e-4ec5-ae50-226bc71789a1\") " Oct 29 23:50:40.904943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.904895 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-socket" pod "ef78a262-e18e-4ec5-ae50-226bc71789a1" (UID: "ef78a262-e18e-4ec5-ae50-226bc71789a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:40.905192 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.905005 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef78a262-e18e-4ec5-ae50-226bc71789a1-istiod-ca-cert" pod "ef78a262-e18e-4ec5-ae50-226bc71789a1" (UID: "ef78a262-e18e-4ec5-ae50-226bc71789a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:40.905257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.905187 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-data" pod "ef78a262-e18e-4ec5-ae50-226bc71789a1" (UID: "ef78a262-e18e-4ec5-ae50-226bc71789a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:40.905257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.905200 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-credential-socket" pod "ef78a262-e18e-4ec5-ae50-226bc71789a1" (UID: "ef78a262-e18e-4ec5-ae50-226bc71789a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:40.905527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.905504 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:40.905604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.905535 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:40.905604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.905552 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:40.905604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.905567 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78a262-e18e-4ec5-ae50-226bc71789a1-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:40.905604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.905597 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:40.906702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.906679 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-envoy" pod "ef78a262-e18e-4ec5-ae50-226bc71789a1" (UID: "ef78a262-e18e-4ec5-ae50-226bc71789a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:40.906911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.906893 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-kube-api-access-tnnfv" pod "ef78a262-e18e-4ec5-ae50-226bc71789a1" (UID: "ef78a262-e18e-4ec5-ae50-226bc71789a1"). InnerVolumeSpecName "kube-api-access-tnnfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:40.907230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.907210 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-podinfo" pod "ef78a262-e18e-4ec5-ae50-226bc71789a1" (UID: "ef78a262-e18e-4ec5-ae50-226bc71789a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:50:40.907300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:40.907216 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-token" pod "ef78a262-e18e-4ec5-ae50-226bc71789a1" (UID: "ef78a262-e18e-4ec5-ae50-226bc71789a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:41.006575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:41.006507 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:41.006575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:41.006536 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnnfv\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-kube-api-access-tnnfv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:41.006575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:41.006547 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:41.006575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:41.006556 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef78a262-e18e-4ec5-ae50-226bc71789a1-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:41.601799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:41.601765 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" Oct 29 23:50:41.602231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:41.601801 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt" event={"ID":"ef78a262-e18e-4ec5-ae50-226bc71789a1","Type":"ContainerDied","Data":"e50aec1ba6122d07a459d9c9ee6c78ad9558548524caefb100afb9656c1b812a"} Oct 29 23:50:41.602231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:41.601858 5736 scope.go:122] "RemoveContainer" containerID="dbe537a4c8b7b3f860000466c566e5bed205f6ce5b6f83b9eb890d1b50f499fb" Oct 29 23:50:41.667638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:41.667610 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt"] Oct 29 23:50:41.670517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:41.670490 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-85hdt"] Oct 29 23:50:43.004969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:43.004936 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" path="/var/lib/kubelet/pods/ef78a262-e18e-4ec5-ae50-226bc71789a1/volumes" Oct 29 23:50:43.930861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:43.930824 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w"] Oct 29 23:50:45.847650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:45.847613 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5"] Oct 29 23:50:45.848099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:45.847866 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" containerID="cri-o://a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed" gracePeriod=30 Oct 29 23:50:46.432394 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.432354 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn"] Oct 29 23:50:46.432991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.432975 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" Oct 29 23:50:46.433046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.432992 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" Oct 29 23:50:46.433046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433013 5736 cpu_manager.go:414] "containerMap: removing container" podUID="64e5d601-93ef-4424-b06c-bc5ca97943e5" containerName="maas-api" Oct 29 23:50:46.433046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433018 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e5d601-93ef-4424-b06c-bc5ca97943e5" containerName="maas-api" Oct 29 23:50:46.433046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433030 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" Oct 29 23:50:46.433046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433036 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433053 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433059 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433070 5736 cpu_manager.go:414] "containerMap: removing container" podUID="351f64d9-561d-49be-a84b-14b91010bcec" containerName="revoke-keys" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433075 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351f64d9-561d-49be-a84b-14b91010bcec" containerName="revoke-keys" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433084 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f7dc7648-76f9-4d85-81ea-ece60b3f76ae" containerName="main" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433089 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7dc7648-76f9-4d85-81ea-ece60b3f76ae" containerName="main" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433100 5736 cpu_manager.go:414] "containerMap: removing container" podUID="45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7" containerName="revoke-keys" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433105 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7" containerName="revoke-keys" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433210 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" Oct 29 23:50:46.433223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433223 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e5d601-93ef-4424-b06c-bc5ca97943e5" containerName="maas-api" Oct 29 23:50:46.433536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433230 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" Oct 29 23:50:46.433536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433238 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b002e9fa-a1b2-4a58-9468-0a6de1a35017" containerName="istio-proxy" Oct 29 23:50:46.433536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433245 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="351f64d9-561d-49be-a84b-14b91010bcec" containerName="revoke-keys" Oct 29 23:50:46.433536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433254 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7dc7648-76f9-4d85-81ea-ece60b3f76ae" containerName="main" Oct 29 23:50:46.433536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.433261 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7" containerName="revoke-keys" Oct 29 23:50:46.438414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.438392 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.457805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.457775 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn"] Oct 29 23:50:46.506802 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.506768 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.506956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.506888 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5z8g\" (UniqueName: \"kubernetes.io/projected/3c56f0ec-5138-47cd-892a-29b8df87075c-kube-api-access-w5z8g\") pod \"maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.507068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.506960 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.512950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.512928 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:50:46.513144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.513126 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:50:46.608392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.608357 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5z8g\" (UniqueName: \"kubernetes.io/projected/3c56f0ec-5138-47cd-892a-29b8df87075c-kube-api-access-w5z8g\") pod \"maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.608520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.608461 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.608669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.608643 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.609179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.609148 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.611036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.611013 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.631640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.631612 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5z8g\" (UniqueName: \"kubernetes.io/projected/3c56f0ec-5138-47cd-892a-29b8df87075c-kube-api-access-w5z8g\") pod \"maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:46.752090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:46.752062 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:47.055726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:47.055437 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn"] Oct 29 23:50:47.059074 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:50:47.059040 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c56f0ec_5138_47cd_892a_29b8df87075c.slice/crio-02500f10ef2699d1e1fdfdfbe74657bda47070c7ccb70248dd6d179fd58a9034 WatchSource:0}: Error finding container 02500f10ef2699d1e1fdfdfbe74657bda47070c7ccb70248dd6d179fd58a9034: Status 404 returned error can't find the container with id 02500f10ef2699d1e1fdfdfbe74657bda47070c7ccb70248dd6d179fd58a9034 Oct 29 23:50:47.674466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:47.674413 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" event={"ID":"3c56f0ec-5138-47cd-892a-29b8df87075c","Type":"ContainerStarted","Data":"b6942a3fd581494cf9da296491f81ed6d226d6de95cc43b3fac852b2db6de5b2"} Oct 29 23:50:47.674638 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:47.674486 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" event={"ID":"3c56f0ec-5138-47cd-892a-29b8df87075c","Type":"ContainerStarted","Data":"02500f10ef2699d1e1fdfdfbe74657bda47070c7ccb70248dd6d179fd58a9034"} Oct 29 23:50:47.715782 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:47.715732 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" podStartSLOduration=1.715720448 podStartE2EDuration="1.715720448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:50:47.715533413 +0000 UTC m=+4044.831453018" watchObservedRunningTime="2026-10-29 23:50:47.715720448 +0000 UTC m=+4044.831640052" Oct 29 23:50:48.685808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:48.685770 5736 generic.go:356] "Generic (PLEG): container finished" podID="3c56f0ec-5138-47cd-892a-29b8df87075c" containerID="b6942a3fd581494cf9da296491f81ed6d226d6de95cc43b3fac852b2db6de5b2" exitCode=0 Oct 29 23:50:48.686246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:48.685865 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" event={"ID":"3c56f0ec-5138-47cd-892a-29b8df87075c","Type":"ContainerDied","Data":"b6942a3fd581494cf9da296491f81ed6d226d6de95cc43b3fac852b2db6de5b2"} Oct 29 23:50:48.931985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:48.931938 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" podUID="9fbc02cd-621e-4251-a0c7-4835484302c7" containerName="payload-processing" containerID="cri-o://59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8" gracePeriod=25 Oct 29 23:50:49.206855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.206802 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:50:49.253381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.253347 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbc02cd-621e-4251-a0c7-4835484302c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc02cd-621e-4251-a0c7-4835484302c7-plugins-config-volume\") pod \"9fbc02cd-621e-4251-a0c7-4835484302c7\" (UID: \"9fbc02cd-621e-4251-a0c7-4835484302c7\") " Oct 29 23:50:49.253554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.253536 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbc02cd-621e-4251-a0c7-4835484302c7-kube-api-access-7mwwn\" (UniqueName: \"kubernetes.io/projected/9fbc02cd-621e-4251-a0c7-4835484302c7-kube-api-access-7mwwn\") pod \"9fbc02cd-621e-4251-a0c7-4835484302c7\" (UID: \"9fbc02cd-621e-4251-a0c7-4835484302c7\") " Oct 29 23:50:49.253730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.253707 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbc02cd-621e-4251-a0c7-4835484302c7-plugins-config-volume" pod "9fbc02cd-621e-4251-a0c7-4835484302c7" (UID: "9fbc02cd-621e-4251-a0c7-4835484302c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:49.254064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.254043 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbc02cd-621e-4251-a0c7-4835484302c7-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:49.255688 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.255669 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbc02cd-621e-4251-a0c7-4835484302c7-kube-api-access-7mwwn" pod "9fbc02cd-621e-4251-a0c7-4835484302c7" (UID: "9fbc02cd-621e-4251-a0c7-4835484302c7"). InnerVolumeSpecName "kube-api-access-7mwwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:49.355283 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.355245 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mwwn\" (UniqueName: \"kubernetes.io/projected/9fbc02cd-621e-4251-a0c7-4835484302c7-kube-api-access-7mwwn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:49.695420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.695388 5736 generic.go:356] "Generic (PLEG): container finished" podID="9fbc02cd-621e-4251-a0c7-4835484302c7" containerID="59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8" exitCode=0 Oct 29 23:50:49.695877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.695434 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" Oct 29 23:50:49.695877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.695468 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" event={"ID":"9fbc02cd-621e-4251-a0c7-4835484302c7","Type":"ContainerDied","Data":"59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8"} Oct 29 23:50:49.695877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.695520 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w" event={"ID":"9fbc02cd-621e-4251-a0c7-4835484302c7","Type":"ContainerDied","Data":"79073c43c39dd251cf66188595546a85f4897f610db5b9e0e08dd9b4290dd02d"} Oct 29 23:50:49.695877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.695542 5736 scope.go:122] "RemoveContainer" containerID="59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8" Oct 29 23:50:49.712934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.712913 5736 scope.go:122] "RemoveContainer" containerID="59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8" Oct 29 23:50:49.714170 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:49.713485 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8\": container with ID starting with 59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8 not found: ID does not exist" containerID="59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8" Oct 29 23:50:49.714170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.713524 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8"} err="failed to get container status \"59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8\": rpc error: code = NotFound desc = could not find container \"59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8\": container with ID starting with 59b2ed74255b949af2a05e22ed2f561af5cd430eab726597c55a051be8b690e8 not found: ID does not exist" Oct 29 23:50:49.786145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.786077 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w"] Oct 29 23:50:49.790537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.790506 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2b0108-597948f556-45g4w"] Oct 29 23:50:49.842515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.842484 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:49.965099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.965071 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c56f0ec-5138-47cd-892a-29b8df87075c-kube-api-access-w5z8g\" (UniqueName: \"kubernetes.io/projected/3c56f0ec-5138-47cd-892a-29b8df87075c-kube-api-access-w5z8g\") pod \"3c56f0ec-5138-47cd-892a-29b8df87075c\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " Oct 29 23:50:49.965280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.965153 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-openshift-ca\") pod \"3c56f0ec-5138-47cd-892a-29b8df87075c\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " Oct 29 23:50:49.965280 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.965225 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-xks-ca\") pod \"3c56f0ec-5138-47cd-892a-29b8df87075c\" (UID: \"3c56f0ec-5138-47cd-892a-29b8df87075c\") " Oct 29 23:50:49.965521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.965498 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-openshift-ca" pod "3c56f0ec-5138-47cd-892a-29b8df87075c" (UID: "3c56f0ec-5138-47cd-892a-29b8df87075c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:49.965742 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.965723 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:49.967151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.967124 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c56f0ec-5138-47cd-892a-29b8df87075c-kube-api-access-w5z8g" pod "3c56f0ec-5138-47cd-892a-29b8df87075c" (UID: "3c56f0ec-5138-47cd-892a-29b8df87075c"). InnerVolumeSpecName "kube-api-access-w5z8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:49.967201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:49.967165 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-xks-ca" pod "3c56f0ec-5138-47cd-892a-29b8df87075c" (UID: "3c56f0ec-5138-47cd-892a-29b8df87075c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:50.066912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:50.066877 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5z8g\" (UniqueName: \"kubernetes.io/projected/3c56f0ec-5138-47cd-892a-29b8df87075c-kube-api-access-w5z8g\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:50.066912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:50.066914 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c56f0ec-5138-47cd-892a-29b8df87075c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:50.705544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:50.705509 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" Oct 29 23:50:50.705965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:50.705541 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn" event={"ID":"3c56f0ec-5138-47cd-892a-29b8df87075c","Type":"ContainerDied","Data":"02500f10ef2699d1e1fdfdfbe74657bda47070c7ccb70248dd6d179fd58a9034"} Oct 29 23:50:50.705965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:50.705605 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02500f10ef2699d1e1fdfdfbe74657bda47070c7ccb70248dd6d179fd58a9034" Oct 29 23:50:51.008829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.008791 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbc02cd-621e-4251-a0c7-4835484302c7" path="/var/lib/kubelet/pods/9fbc02cd-621e-4251-a0c7-4835484302c7/volumes" Oct 29 23:50:51.126127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.126098 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:50:51.184124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184098 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-socket\") pod \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " Oct 29 23:50:51.184288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184148 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-podinfo\") pod \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " Oct 29 23:50:51.184288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184181 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-certs\") pod \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " Oct 29 23:50:51.184288 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184221 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-kube-api-access-6q6dn\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-kube-api-access-6q6dn\") pod \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " Oct 29 23:50:51.184467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184330 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-credential-socket\") pod \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " Oct 29 23:50:51.184467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184371 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-socket" pod "d69cbd5a-5121-4051-ab5c-7ef57eda66c4" (UID: "d69cbd5a-5121-4051-ab5c-7ef57eda66c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:51.184467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184442 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-envoy\") pod \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " Oct 29 23:50:51.184696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184483 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istiod-ca-cert\") pod \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " Oct 29 23:50:51.184696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184494 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-certs" pod "d69cbd5a-5121-4051-ab5c-7ef57eda66c4" (UID: "d69cbd5a-5121-4051-ab5c-7ef57eda66c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:51.184696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184513 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-token\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-token\") pod \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " Oct 29 23:50:51.184696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184628 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-data\") pod \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\" (UID: \"d69cbd5a-5121-4051-ab5c-7ef57eda66c4\") " Oct 29 23:50:51.184937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184739 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-credential-socket" pod "d69cbd5a-5121-4051-ab5c-7ef57eda66c4" (UID: "d69cbd5a-5121-4051-ab5c-7ef57eda66c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:51.184937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.184843 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istiod-ca-cert" pod "d69cbd5a-5121-4051-ab5c-7ef57eda66c4" (UID: "d69cbd5a-5121-4051-ab5c-7ef57eda66c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:51.185231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.185110 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-data" pod "d69cbd5a-5121-4051-ab5c-7ef57eda66c4" (UID: "d69cbd5a-5121-4051-ab5c-7ef57eda66c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:51.186446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.186397 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:51.186446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.186427 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:51.186446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.186443 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:51.187122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.186458 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:51.187122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.186471 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:51.187761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.187738 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-token" pod "d69cbd5a-5121-4051-ab5c-7ef57eda66c4" (UID: "d69cbd5a-5121-4051-ab5c-7ef57eda66c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:51.187862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.187774 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-kube-api-access-6q6dn" pod "d69cbd5a-5121-4051-ab5c-7ef57eda66c4" (UID: "d69cbd5a-5121-4051-ab5c-7ef57eda66c4"). InnerVolumeSpecName "kube-api-access-6q6dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:51.188665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.188502 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-podinfo" pod "d69cbd5a-5121-4051-ab5c-7ef57eda66c4" (UID: "d69cbd5a-5121-4051-ab5c-7ef57eda66c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:50:51.188665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.188620 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-envoy" pod "d69cbd5a-5121-4051-ab5c-7ef57eda66c4" (UID: "d69cbd5a-5121-4051-ab5c-7ef57eda66c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:50:51.287821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.287750 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:51.287821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.287777 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:51.287821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.287787 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:51.287821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.287796 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6dn\" (UniqueName: \"kubernetes.io/projected/d69cbd5a-5121-4051-ab5c-7ef57eda66c4-kube-api-access-6q6dn\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:51.717405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.717365 5736 generic.go:356] "Generic (PLEG): container finished" podID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerID="a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed" exitCode=0 Oct 29 23:50:51.717866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.717410 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" Oct 29 23:50:51.717866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.717439 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" event={"ID":"d69cbd5a-5121-4051-ab5c-7ef57eda66c4","Type":"ContainerDied","Data":"a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed"} Oct 29 23:50:51.717866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.717503 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5" event={"ID":"d69cbd5a-5121-4051-ab5c-7ef57eda66c4","Type":"ContainerDied","Data":"68d066e0b5d9963923c655f9fdec58fa5c8ccee625992adad92d3d928fe47cf7"} Oct 29 23:50:51.717866 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.717536 5736 scope.go:122] "RemoveContainer" containerID="a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed" Oct 29 23:50:51.731296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.731277 5736 scope.go:122] "RemoveContainer" containerID="d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230" Oct 29 23:50:51.741258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.741240 5736 scope.go:122] "RemoveContainer" containerID="a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed" Oct 29 23:50:51.741491 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:51.741470 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed\": container with ID starting with a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed not found: ID does not exist" containerID="a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed" Oct 29 23:50:51.741543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.741500 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed"} err="failed to get container status \"a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed\": rpc error: code = NotFound desc = could not find container \"a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed\": container with ID starting with a7e3b604ae5fe73ae2a487940f2ba109298e7fb2466dfdbc4b9fa21c42d558ed not found: ID does not exist" Oct 29 23:50:51.741543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.741517 5736 scope.go:122] "RemoveContainer" containerID="d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230" Oct 29 23:50:51.741778 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:51.741759 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230\": container with ID starting with d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230 not found: ID does not exist" containerID="d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230" Oct 29 23:50:51.741819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.741782 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230"} err="failed to get container status \"d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230\": rpc error: code = NotFound desc = could not find container \"d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230\": container with ID starting with d66dd4cc6c4caf7990fcfe962dbbe6e7a4c97fc2f096389457ab2d38bc543230 not found: ID does not exist" Oct 29 23:50:51.783436 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.783407 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5"] Oct 29 23:50:51.793914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:51.793893 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-750972-openshift-default-6cd6dd9687-fwbc5"] Oct 29 23:50:52.836313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.836265 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp"] Oct 29 23:50:52.837101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837078 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" Oct 29 23:50:52.837101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837101 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" Oct 29 23:50:52.837255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837137 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9fbc02cd-621e-4251-a0c7-4835484302c7" containerName="payload-processing" Oct 29 23:50:52.837255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837146 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbc02cd-621e-4251-a0c7-4835484302c7" containerName="payload-processing" Oct 29 23:50:52.837255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837157 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" Oct 29 23:50:52.837255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837162 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" Oct 29 23:50:52.837255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837181 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3c56f0ec-5138-47cd-892a-29b8df87075c" containerName="revoke-keys" Oct 29 23:50:52.837255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837187 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c56f0ec-5138-47cd-892a-29b8df87075c" containerName="revoke-keys" Oct 29 23:50:52.837575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837274 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c56f0ec-5138-47cd-892a-29b8df87075c" containerName="revoke-keys" Oct 29 23:50:52.837575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837282 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" Oct 29 23:50:52.837575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837292 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" Oct 29 23:50:52.837575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837302 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef78a262-e18e-4ec5-ae50-226bc71789a1" containerName="istio-proxy" Oct 29 23:50:52.837575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.837310 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbc02cd-621e-4251-a0c7-4835484302c7" containerName="payload-processing" Oct 29 23:50:52.842442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.842413 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:52.869861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.869807 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-2b0108-openshift-default-dockercfg-v8tck\"" Oct 29 23:50:52.873899 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.873872 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp"] Oct 29 23:50:52.909498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.909465 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:52.909657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.909531 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q67vm\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-kube-api-access-q67vm\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:52.909657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.909621 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:52.909657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.909649 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:52.909798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.909760 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:52.910212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.909982 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:52.910212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.910024 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:52.910212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.910132 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:52.910464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:52.910212 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.004104 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.004064 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" path="/var/lib/kubelet/pods/d69cbd5a-5121-4051-ab5c-7ef57eda66c4/volumes" Oct 29 23:50:53.011880 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.011851 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.011907 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q67vm\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-kube-api-access-q67vm\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.011952 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.011981 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012016 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012109 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012135 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012201 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012254 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012406 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012420 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012499 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012570 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.012919 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.012896 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.014413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.014394 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.014605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.014571 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.037806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.037775 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.037806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.037799 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q67vm\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-kube-api-access-q67vm\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.158368 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.158295 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:53.489045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.489015 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp"] Oct 29 23:50:53.492950 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:50:53.492916 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc17b25_421e_405d_ba4f_ebd1af1cdc0b.slice/crio-4f69e333464158be7a17b1ddba0d18ce112eb9f122e711a911a7caa5e6bbf0cf WatchSource:0}: Error finding container 4f69e333464158be7a17b1ddba0d18ce112eb9f122e711a911a7caa5e6bbf0cf: Status 404 returned error can't find the container with id 4f69e333464158be7a17b1ddba0d18ce112eb9f122e711a911a7caa5e6bbf0cf Oct 29 23:50:53.495665 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.495630 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:50:53.495768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.495699 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:50:53.495768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.495727 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:50:53.738450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.738410 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" event={"ID":"efc17b25-421e-405d-ba4f-ebd1af1cdc0b","Type":"ContainerStarted","Data":"c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b"} Oct 29 23:50:53.738687 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.738465 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" event={"ID":"efc17b25-421e-405d-ba4f-ebd1af1cdc0b","Type":"ContainerStarted","Data":"4f69e333464158be7a17b1ddba0d18ce112eb9f122e711a911a7caa5e6bbf0cf"} Oct 29 23:50:53.758916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.758878 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f"] Oct 29 23:50:53.759506 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.759490 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" Oct 29 23:50:53.759549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.759508 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69cbd5a-5121-4051-ab5c-7ef57eda66c4" containerName="istio-proxy" Oct 29 23:50:53.763132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.763110 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:53.780051 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.780028 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:50:53.780175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.780086 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:50:53.798890 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.798765 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f"] Oct 29 23:50:53.801646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.801573 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" podStartSLOduration=1.8015580770000001 podStartE2EDuration="1.801558077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:50:53.798488871 +0000 UTC m=+4050.914408480" watchObservedRunningTime="2026-10-29 23:50:53.801558077 +0000 UTC m=+4050.917477686" Oct 29 23:50:53.924326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.924291 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98blx\" (UniqueName: \"kubernetes.io/projected/145e43bb-0faa-4488-8aff-4c3ae43208c1-kube-api-access-98blx\") pod \"maas-api-revoke-sub-82dade27-c2t5f\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:53.924740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.924484 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82dade27-c2t5f\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:53.924740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:53.924546 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82dade27-c2t5f\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:54.025987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.025910 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98blx\" (UniqueName: \"kubernetes.io/projected/145e43bb-0faa-4488-8aff-4c3ae43208c1-kube-api-access-98blx\") pod \"maas-api-revoke-sub-82dade27-c2t5f\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:54.026137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.026051 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82dade27-c2t5f\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:54.026137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.026097 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82dade27-c2t5f\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:54.026756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.026729 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82dade27-c2t5f\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:54.028438 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.028413 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82dade27-c2t5f\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:54.049593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.049554 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98blx\" (UniqueName: \"kubernetes.io/projected/145e43bb-0faa-4488-8aff-4c3ae43208c1-kube-api-access-98blx\") pod \"maas-api-revoke-sub-82dade27-c2t5f\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:54.078530 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.078506 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:54.159314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.159281 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:50:54.160642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.160612 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" start-of-body= Oct 29 23:50:54.160774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.160670 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" Oct 29 23:50:54.384374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.384347 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f"] Oct 29 23:50:54.387116 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:50:54.387086 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145e43bb_0faa_4488_8aff_4c3ae43208c1.slice/crio-350a6f6d88c65f3366866e384c020a26333d3e37a7ea8ff166720f78e9ff6cbd WatchSource:0}: Error finding container 350a6f6d88c65f3366866e384c020a26333d3e37a7ea8ff166720f78e9ff6cbd: Status 404 returned error can't find the container with id 350a6f6d88c65f3366866e384c020a26333d3e37a7ea8ff166720f78e9ff6cbd Oct 29 23:50:54.764861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:54.764823 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" event={"ID":"145e43bb-0faa-4488-8aff-4c3ae43208c1","Type":"ContainerStarted","Data":"350a6f6d88c65f3366866e384c020a26333d3e37a7ea8ff166720f78e9ff6cbd"} Oct 29 23:50:55.159599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:55.159478 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" start-of-body= Oct 29 23:50:55.160203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:55.160162 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" Oct 29 23:50:55.774797 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:55.774761 5736 generic.go:356] "Generic (PLEG): container finished" podID="145e43bb-0faa-4488-8aff-4c3ae43208c1" containerID="a260acd4b1fe35f6562ed969939d561b0bdd158508f859f80f5c9a84362fb5d2" exitCode=0 Oct 29 23:50:55.774982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:55.774837 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" event={"ID":"145e43bb-0faa-4488-8aff-4c3ae43208c1","Type":"ContainerDied","Data":"a260acd4b1fe35f6562ed969939d561b0bdd158508f859f80f5c9a84362fb5d2"} Oct 29 23:50:56.158846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:56.158747 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" start-of-body= Oct 29 23:50:56.158846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:56.158810 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" Oct 29 23:50:56.826596 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:50:56.826541 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:50:56.984233 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:56.984198 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:57.088822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.088742 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145e43bb-0faa-4488-8aff-4c3ae43208c1-kube-api-access-98blx\" (UniqueName: \"kubernetes.io/projected/145e43bb-0faa-4488-8aff-4c3ae43208c1-kube-api-access-98blx\") pod \"145e43bb-0faa-4488-8aff-4c3ae43208c1\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " Oct 29 23:50:57.088977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.088926 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-openshift-ca\") pod \"145e43bb-0faa-4488-8aff-4c3ae43208c1\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " Oct 29 23:50:57.089023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.088988 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-xks-ca\") pod \"145e43bb-0faa-4488-8aff-4c3ae43208c1\" (UID: \"145e43bb-0faa-4488-8aff-4c3ae43208c1\") " Oct 29 23:50:57.090774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.090732 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-openshift-ca" pod "145e43bb-0faa-4488-8aff-4c3ae43208c1" (UID: "145e43bb-0faa-4488-8aff-4c3ae43208c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:50:57.091737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.091711 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-xks-ca" pod "145e43bb-0faa-4488-8aff-4c3ae43208c1" (UID: "145e43bb-0faa-4488-8aff-4c3ae43208c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:50:57.093316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.093275 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145e43bb-0faa-4488-8aff-4c3ae43208c1-kube-api-access-98blx" pod "145e43bb-0faa-4488-8aff-4c3ae43208c1" (UID: "145e43bb-0faa-4488-8aff-4c3ae43208c1"). InnerVolumeSpecName "kube-api-access-98blx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:50:57.161479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.160993 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" start-of-body= Oct 29 23:50:57.161479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.161061 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" Oct 29 23:50:57.191323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.191254 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98blx\" (UniqueName: \"kubernetes.io/projected/145e43bb-0faa-4488-8aff-4c3ae43208c1-kube-api-access-98blx\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:57.191323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.191288 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:57.191323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.191301 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145e43bb-0faa-4488-8aff-4c3ae43208c1-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:50:57.791781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.791749 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" Oct 29 23:50:57.791991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.791788 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f" event={"ID":"145e43bb-0faa-4488-8aff-4c3ae43208c1","Type":"ContainerDied","Data":"350a6f6d88c65f3366866e384c020a26333d3e37a7ea8ff166720f78e9ff6cbd"} Oct 29 23:50:57.791991 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:57.791839 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="350a6f6d88c65f3366866e384c020a26333d3e37a7ea8ff166720f78e9ff6cbd" Oct 29 23:50:58.159251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:58.159172 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" start-of-body= Oct 29 23:50:58.159715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:58.159679 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" Oct 29 23:50:59.159276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:59.159236 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" start-of-body= Oct 29 23:50:59.159778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:50:59.159295 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" Oct 29 23:51:00.158937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:00.158898 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" start-of-body= Oct 29 23:51:00.159097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:00.158964 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.188:15021/healthz/ready\": dial tcp 10.134.1.188:15021: connect: connection refused" Oct 29 23:51:01.163304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.163272 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:51:01.163872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.163804 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:51:01.164509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.164476 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:51:01.619399 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.619366 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td"] Oct 29 23:51:01.620057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.620030 5736 cpu_manager.go:414] "containerMap: removing container" podUID="145e43bb-0faa-4488-8aff-4c3ae43208c1" containerName="revoke-keys" Oct 29 23:51:01.620057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.620051 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145e43bb-0faa-4488-8aff-4c3ae43208c1" containerName="revoke-keys" Oct 29 23:51:01.620204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.620151 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="145e43bb-0faa-4488-8aff-4c3ae43208c1" containerName="revoke-keys" Oct 29 23:51:01.623499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.623483 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.638286 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.638264 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:51:01.638392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.638379 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:51:01.645477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.645455 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td"] Oct 29 23:51:01.678281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.678247 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ecfa1a-cs7td\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.678441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.678319 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9kzs\" (UniqueName: \"kubernetes.io/projected/7df8051e-1cd4-45dc-b68e-d4d7875eade9-kube-api-access-z9kzs\") pod \"maas-api-revoke-sub-38ecfa1a-cs7td\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.678549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.678521 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ecfa1a-cs7td\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.779531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.779490 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ecfa1a-cs7td\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.779752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.779664 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ecfa1a-cs7td\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.779752 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.779706 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kzs\" (UniqueName: \"kubernetes.io/projected/7df8051e-1cd4-45dc-b68e-d4d7875eade9-kube-api-access-z9kzs\") pod \"maas-api-revoke-sub-38ecfa1a-cs7td\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.780322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.780300 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ecfa1a-cs7td\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.782122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.782090 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ecfa1a-cs7td\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.803058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.803025 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kzs\" (UniqueName: \"kubernetes.io/projected/7df8051e-1cd4-45dc-b68e-d4d7875eade9-kube-api-access-z9kzs\") pod \"maas-api-revoke-sub-38ecfa1a-cs7td\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:01.938011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:01.937929 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:02.247516 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:02.247484 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td"] Oct 29 23:51:02.250800 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:51:02.250758 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df8051e_1cd4_45dc_b68e_d4d7875eade9.slice/crio-d0313d16e8eb85959da15525559e13b90cd4c2916e404fe7c6352f8e5da6ad64 WatchSource:0}: Error finding container d0313d16e8eb85959da15525559e13b90cd4c2916e404fe7c6352f8e5da6ad64: Status 404 returned error can't find the container with id d0313d16e8eb85959da15525559e13b90cd4c2916e404fe7c6352f8e5da6ad64 Oct 29 23:51:02.834699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:02.834603 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" event={"ID":"7df8051e-1cd4-45dc-b68e-d4d7875eade9","Type":"ContainerStarted","Data":"7e4318ec02f6bf8fc94e3b31899eae41e97281ab340ebc032bd0da15c86a5295"} Oct 29 23:51:02.834699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:02.834664 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" event={"ID":"7df8051e-1cd4-45dc-b68e-d4d7875eade9","Type":"ContainerStarted","Data":"d0313d16e8eb85959da15525559e13b90cd4c2916e404fe7c6352f8e5da6ad64"} Oct 29 23:51:02.875977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:02.875927 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" podStartSLOduration=1.8759149320000001 podStartE2EDuration="1.875914932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:51:02.875606864 +0000 UTC m=+4059.991526463" watchObservedRunningTime="2026-10-29 23:51:02.875914932 +0000 UTC m=+4059.991834537" Oct 29 23:51:03.843844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:03.843751 5736 generic.go:356] "Generic (PLEG): container finished" podID="7df8051e-1cd4-45dc-b68e-d4d7875eade9" containerID="7e4318ec02f6bf8fc94e3b31899eae41e97281ab340ebc032bd0da15c86a5295" exitCode=0 Oct 29 23:51:03.844287 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:03.843836 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" event={"ID":"7df8051e-1cd4-45dc-b68e-d4d7875eade9","Type":"ContainerDied","Data":"7e4318ec02f6bf8fc94e3b31899eae41e97281ab340ebc032bd0da15c86a5295"} Oct 29 23:51:05.038468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.038430 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst"] Oct 29 23:51:05.044498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.044471 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.060944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.060917 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst"] Oct 29 23:51:05.096589 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.096556 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:05.131884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.131823 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b8149-5plst\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.132065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.132037 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w9w2\" (UniqueName: \"kubernetes.io/projected/60e6eb7e-a38e-4e4f-8104-d049839bbcab-kube-api-access-7w9w2\") pod \"maas-api-revoke-keys-e2e-inf-a-3b8149-5plst\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.132148 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.132110 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b8149-5plst\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.233308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.233273 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-openshift-ca\") pod \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " Oct 29 23:51:05.233496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.233362 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df8051e-1cd4-45dc-b68e-d4d7875eade9-kube-api-access-z9kzs\" (UniqueName: \"kubernetes.io/projected/7df8051e-1cd4-45dc-b68e-d4d7875eade9-kube-api-access-z9kzs\") pod \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " Oct 29 23:51:05.233496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.233479 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-xks-ca\") pod \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\" (UID: \"7df8051e-1cd4-45dc-b68e-d4d7875eade9\") " Oct 29 23:51:05.233771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.233745 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b8149-5plst\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.233863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.233762 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-openshift-ca" pod "7df8051e-1cd4-45dc-b68e-d4d7875eade9" (UID: "7df8051e-1cd4-45dc-b68e-d4d7875eade9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:05.233941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.233871 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b8149-5plst\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.234107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.234088 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9w2\" (UniqueName: \"kubernetes.io/projected/60e6eb7e-a38e-4e4f-8104-d049839bbcab-kube-api-access-7w9w2\") pod \"maas-api-revoke-keys-e2e-inf-a-3b8149-5plst\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.235240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.234324 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:05.235240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.234493 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b8149-5plst\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.235820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.235794 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df8051e-1cd4-45dc-b68e-d4d7875eade9-kube-api-access-z9kzs" pod "7df8051e-1cd4-45dc-b68e-d4d7875eade9" (UID: "7df8051e-1cd4-45dc-b68e-d4d7875eade9"). InnerVolumeSpecName "kube-api-access-z9kzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:05.235974 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.235952 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-xks-ca" pod "7df8051e-1cd4-45dc-b68e-d4d7875eade9" (UID: "7df8051e-1cd4-45dc-b68e-d4d7875eade9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:05.236313 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.236293 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b8149-5plst\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.256813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.256792 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9w2\" (UniqueName: \"kubernetes.io/projected/60e6eb7e-a38e-4e4f-8104-d049839bbcab-kube-api-access-7w9w2\") pod \"maas-api-revoke-keys-e2e-inf-a-3b8149-5plst\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.335716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.335678 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9kzs\" (UniqueName: \"kubernetes.io/projected/7df8051e-1cd4-45dc-b68e-d4d7875eade9-kube-api-access-z9kzs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:05.335716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.335716 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df8051e-1cd4-45dc-b68e-d4d7875eade9-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:05.360453 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.360407 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:05.673337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.673304 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst"] Oct 29 23:51:05.675744 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:51:05.675706 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e6eb7e_a38e_4e4f_8104_d049839bbcab.slice/crio-fe0f157c7c6a749fa75cb46c7d1c23e2b1076844b66ff9a1ef9f80c469d9ed5c WatchSource:0}: Error finding container fe0f157c7c6a749fa75cb46c7d1c23e2b1076844b66ff9a1ef9f80c469d9ed5c: Status 404 returned error can't find the container with id fe0f157c7c6a749fa75cb46c7d1c23e2b1076844b66ff9a1ef9f80c469d9ed5c Oct 29 23:51:05.869252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.869215 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" event={"ID":"60e6eb7e-a38e-4e4f-8104-d049839bbcab","Type":"ContainerStarted","Data":"fe0f157c7c6a749fa75cb46c7d1c23e2b1076844b66ff9a1ef9f80c469d9ed5c"} Oct 29 23:51:05.870692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.870668 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" Oct 29 23:51:05.870806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.870693 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td" event={"ID":"7df8051e-1cd4-45dc-b68e-d4d7875eade9","Type":"ContainerDied","Data":"d0313d16e8eb85959da15525559e13b90cd4c2916e404fe7c6352f8e5da6ad64"} Oct 29 23:51:05.870806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:05.870735 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0313d16e8eb85959da15525559e13b90cd4c2916e404fe7c6352f8e5da6ad64" Oct 29 23:51:06.879551 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:06.879474 5736 generic.go:356] "Generic (PLEG): container finished" podID="60e6eb7e-a38e-4e4f-8104-d049839bbcab" containerID="712963b91d66ee15e473ee21c8fa43b1accc7dded6460da68c0a06d330e33157" exitCode=0 Oct 29 23:51:06.879949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:06.879551 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" event={"ID":"60e6eb7e-a38e-4e4f-8104-d049839bbcab","Type":"ContainerDied","Data":"712963b91d66ee15e473ee21c8fa43b1accc7dded6460da68c0a06d330e33157"} Oct 29 23:51:08.075364 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.075331 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:08.180441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.180348 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e6eb7e-a38e-4e4f-8104-d049839bbcab-kube-api-access-7w9w2\" (UniqueName: \"kubernetes.io/projected/60e6eb7e-a38e-4e4f-8104-d049839bbcab-kube-api-access-7w9w2\") pod \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " Oct 29 23:51:08.180441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.180414 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-xks-ca\") pod \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " Oct 29 23:51:08.182238 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.180598 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-openshift-ca\") pod \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\" (UID: \"60e6eb7e-a38e-4e4f-8104-d049839bbcab\") " Oct 29 23:51:08.189412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.184638 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-openshift-ca" pod "60e6eb7e-a38e-4e4f-8104-d049839bbcab" (UID: "60e6eb7e-a38e-4e4f-8104-d049839bbcab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:08.189412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.188477 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-xks-ca" pod "60e6eb7e-a38e-4e4f-8104-d049839bbcab" (UID: "60e6eb7e-a38e-4e4f-8104-d049839bbcab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:08.190941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.190897 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e6eb7e-a38e-4e4f-8104-d049839bbcab-kube-api-access-7w9w2" pod "60e6eb7e-a38e-4e4f-8104-d049839bbcab" (UID: "60e6eb7e-a38e-4e4f-8104-d049839bbcab"). InnerVolumeSpecName "kube-api-access-7w9w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:08.282937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.282888 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w9w2\" (UniqueName: \"kubernetes.io/projected/60e6eb7e-a38e-4e4f-8104-d049839bbcab-kube-api-access-7w9w2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:08.282937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.282936 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:08.282937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.282948 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e6eb7e-a38e-4e4f-8104-d049839bbcab-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:08.896868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.896836 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" Oct 29 23:51:08.897045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.896838 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst" event={"ID":"60e6eb7e-a38e-4e4f-8104-d049839bbcab","Type":"ContainerDied","Data":"fe0f157c7c6a749fa75cb46c7d1c23e2b1076844b66ff9a1ef9f80c469d9ed5c"} Oct 29 23:51:08.897045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:08.896963 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe0f157c7c6a749fa75cb46c7d1c23e2b1076844b66ff9a1ef9f80c469d9ed5c" Oct 29 23:51:17.591422 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:17.591381 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95"] Oct 29 23:51:17.602792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:17.602761 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2b0108-6tr95"] Oct 29 23:51:19.005102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:19.005067 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7" path="/var/lib/kubelet/pods/45a19c33-3b10-4bce-ba8f-3fa01ab9a5c7/volumes" Oct 29 23:51:22.480631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:22.480562 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49"] Oct 29 23:51:22.968729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:22.968695 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr"] Oct 29 23:51:22.969498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:22.969468 5736 cpu_manager.go:414] "containerMap: removing container" podUID="60e6eb7e-a38e-4e4f-8104-d049839bbcab" containerName="revoke-keys" Oct 29 23:51:22.969498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:22.969493 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e6eb7e-a38e-4e4f-8104-d049839bbcab" containerName="revoke-keys" Oct 29 23:51:22.969670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:22.969521 5736 cpu_manager.go:414] "containerMap: removing container" podUID="7df8051e-1cd4-45dc-b68e-d4d7875eade9" containerName="revoke-keys" Oct 29 23:51:22.969670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:22.969526 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df8051e-1cd4-45dc-b68e-d4d7875eade9" containerName="revoke-keys" Oct 29 23:51:22.969743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:22.969669 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df8051e-1cd4-45dc-b68e-d4d7875eade9" containerName="revoke-keys" Oct 29 23:51:22.969743 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:22.969687 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e6eb7e-a38e-4e4f-8104-d049839bbcab" containerName="revoke-keys" Oct 29 23:51:22.974707 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:22.974687 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.014185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.014148 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr"] Oct 29 23:51:23.081049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.081011 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf"] Oct 29 23:51:23.085209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.085187 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.099738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.099716 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:51:23.107604 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.107565 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf"] Oct 29 23:51:23.130748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.130717 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.130902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.130776 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.130971 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.130938 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.131036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.131014 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.131377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.131255 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.131490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.131383 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.131490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.131443 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265571ba-40d8-4fa9-92dd-162d252f09a4-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.131677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.131505 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.131677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.131561 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhvcl\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-kube-api-access-nhvcl\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.200094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.200064 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:51:23.232930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.232855 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.232930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.232917 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c87936-kn9nf\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.233126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.232974 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.233126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233055 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265571ba-40d8-4fa9-92dd-162d252f09a4-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.233126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233117 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.233339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233164 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhvcl\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-kube-api-access-nhvcl\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.233339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233212 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.233339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233241 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5xr6\" (UniqueName: \"kubernetes.io/projected/bef6171a-064f-4f9a-a65d-6fed44a00d36-kube-api-access-f5xr6\") pod \"maas-api-revoke-sub-89c87936-kn9nf\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.233339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233281 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.233339 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233311 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c87936-kn9nf\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.233751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233352 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.233751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233398 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.233751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233452 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.233751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233716 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.234012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233793 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.234012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233861 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.234012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.233913 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265571ba-40d8-4fa9-92dd-162d252f09a4-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.235460 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.235436 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.235710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.235686 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.255926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.255904 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.256358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.256335 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhvcl\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-kube-api-access-nhvcl\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.291659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.291631 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:23.334969 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.334936 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c87936-kn9nf\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.335105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.335077 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5xr6\" (UniqueName: \"kubernetes.io/projected/bef6171a-064f-4f9a-a65d-6fed44a00d36-kube-api-access-f5xr6\") pod \"maas-api-revoke-sub-89c87936-kn9nf\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.335188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.335127 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c87936-kn9nf\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.335833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.335806 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c87936-kn9nf\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.348699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.348676 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:51:23.358027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.358003 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c87936-kn9nf\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.358875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.358856 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5xr6\" (UniqueName: \"kubernetes.io/projected/bef6171a-064f-4f9a-a65d-6fed44a00d36-kube-api-access-f5xr6\") pod \"maas-api-revoke-sub-89c87936-kn9nf\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.413120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.413084 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:51:23.420047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.419666 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:23.625986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.625823 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr"] Oct 29 23:51:23.628818 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:51:23.628786 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265571ba_40d8_4fa9_92dd_162d252f09a4.slice/crio-4b7f3b71ab3ae8ac4bb678b20e00c56d0284d492960dc788a7c0ebbb84b69ff9 WatchSource:0}: Error finding container 4b7f3b71ab3ae8ac4bb678b20e00c56d0284d492960dc788a7c0ebbb84b69ff9: Status 404 returned error can't find the container with id 4b7f3b71ab3ae8ac4bb678b20e00c56d0284d492960dc788a7c0ebbb84b69ff9 Oct 29 23:51:23.631648 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.631612 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:51:23.631726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.631679 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:51:23.631726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.631708 5736 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:51:23.765593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:23.765431 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf"] Oct 29 23:51:23.768930 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:51:23.768903 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef6171a_064f_4f9a_a65d_6fed44a00d36.slice/crio-fc5b944119c72dfd5ce4c24b9da0c97d2072099dae1f6244ff31fa5ff1669a49 WatchSource:0}: Error finding container fc5b944119c72dfd5ce4c24b9da0c97d2072099dae1f6244ff31fa5ff1669a49: Status 404 returned error can't find the container with id fc5b944119c72dfd5ce4c24b9da0c97d2072099dae1f6244ff31fa5ff1669a49 Oct 29 23:51:24.027302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:24.027207 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" event={"ID":"bef6171a-064f-4f9a-a65d-6fed44a00d36","Type":"ContainerStarted","Data":"fc5b944119c72dfd5ce4c24b9da0c97d2072099dae1f6244ff31fa5ff1669a49"} Oct 29 23:51:24.029198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:24.029161 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" event={"ID":"265571ba-40d8-4fa9-92dd-162d252f09a4","Type":"ContainerStarted","Data":"84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af"} Oct 29 23:51:24.029343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:24.029218 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" event={"ID":"265571ba-40d8-4fa9-92dd-162d252f09a4","Type":"ContainerStarted","Data":"4b7f3b71ab3ae8ac4bb678b20e00c56d0284d492960dc788a7c0ebbb84b69ff9"} Oct 29 23:51:24.093335 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:24.091752 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" podStartSLOduration=2.09173498 podStartE2EDuration="2.09173498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:51:24.075633171 +0000 UTC m=+4081.191552778" watchObservedRunningTime="2026-10-29 23:51:24.09173498 +0000 UTC m=+4081.207654583" Oct 29 23:51:24.292717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:24.292638 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:24.294229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:24.294202 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" start-of-body= Oct 29 23:51:24.294338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:24.294250 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" Oct 29 23:51:25.060749 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:25.060652 5736 generic.go:356] "Generic (PLEG): container finished" podID="bef6171a-064f-4f9a-a65d-6fed44a00d36" containerID="9a137bb5e483abefa43d9823719ed903b3d6d206946a53e9f99631d9d396957e" exitCode=0 Oct 29 23:51:25.061234 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:25.060736 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" event={"ID":"bef6171a-064f-4f9a-a65d-6fed44a00d36","Type":"ContainerDied","Data":"9a137bb5e483abefa43d9823719ed903b3d6d206946a53e9f99631d9d396957e"} Oct 29 23:51:25.297636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:25.293828 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" start-of-body= Oct 29 23:51:25.297636 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:25.293940 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" Oct 29 23:51:26.270245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.270212 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:26.292176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.292143 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" start-of-body= Oct 29 23:51:26.292307 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.292193 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" Oct 29 23:51:26.401520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.401428 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-xks-ca\") pod \"bef6171a-064f-4f9a-a65d-6fed44a00d36\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " Oct 29 23:51:26.401727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.401633 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef6171a-064f-4f9a-a65d-6fed44a00d36-kube-api-access-f5xr6\" (UniqueName: \"kubernetes.io/projected/bef6171a-064f-4f9a-a65d-6fed44a00d36-kube-api-access-f5xr6\") pod \"bef6171a-064f-4f9a-a65d-6fed44a00d36\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " Oct 29 23:51:26.401727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.401696 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-openshift-ca\") pod \"bef6171a-064f-4f9a-a65d-6fed44a00d36\" (UID: \"bef6171a-064f-4f9a-a65d-6fed44a00d36\") " Oct 29 23:51:26.402719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.402690 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-openshift-ca" pod "bef6171a-064f-4f9a-a65d-6fed44a00d36" (UID: "bef6171a-064f-4f9a-a65d-6fed44a00d36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:26.404150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.404114 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-xks-ca" pod "bef6171a-064f-4f9a-a65d-6fed44a00d36" (UID: "bef6171a-064f-4f9a-a65d-6fed44a00d36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:26.404849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.404827 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef6171a-064f-4f9a-a65d-6fed44a00d36-kube-api-access-f5xr6" pod "bef6171a-064f-4f9a-a65d-6fed44a00d36" (UID: "bef6171a-064f-4f9a-a65d-6fed44a00d36"). InnerVolumeSpecName "kube-api-access-f5xr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:26.503535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.503501 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:26.503535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.503531 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5xr6\" (UniqueName: \"kubernetes.io/projected/bef6171a-064f-4f9a-a65d-6fed44a00d36-kube-api-access-f5xr6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:26.503535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:26.503541 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef6171a-064f-4f9a-a65d-6fed44a00d36-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:26.824237 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:51:26.824201 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:51:27.080498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:27.080412 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" Oct 29 23:51:27.080498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:27.080447 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf" event={"ID":"bef6171a-064f-4f9a-a65d-6fed44a00d36","Type":"ContainerDied","Data":"fc5b944119c72dfd5ce4c24b9da0c97d2072099dae1f6244ff31fa5ff1669a49"} Oct 29 23:51:27.080727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:27.080508 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5b944119c72dfd5ce4c24b9da0c97d2072099dae1f6244ff31fa5ff1669a49" Oct 29 23:51:27.293071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:27.293030 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" start-of-body= Oct 29 23:51:27.293792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:27.293750 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" Oct 29 23:51:28.293022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:28.292979 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" start-of-body= Oct 29 23:51:28.293216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:28.293050 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" Oct 29 23:51:29.292248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:29.292207 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" start-of-body= Oct 29 23:51:29.292462 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:29.292277 5736 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.192:15021/healthz/ready\": dial tcp 10.134.1.192:15021: connect: connection refused" Oct 29 23:51:30.297457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:30.297423 5736 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:30.297970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:30.297835 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:30.298624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:30.298569 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:31.359388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:31.359351 5736 scope.go:122] "RemoveContainer" containerID="67a452701f2a04310723678c4661970ab11d0d3a99443b2a003a2f00d93bdb54" Oct 29 23:51:31.374147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:31.374121 5736 scope.go:122] "RemoveContainer" containerID="ab7f2056fb9dcae88a6cccd64c16099511bf347f5bec45164e76da2ef6e2525a" Oct 29 23:51:31.384187 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:31.384159 5736 scope.go:122] "RemoveContainer" containerID="dd5adbcb5fe5786c8b14553cde8202328abe98734556b51c0ff94f9477f02e37" Oct 29 23:51:31.393314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:31.393294 5736 scope.go:122] "RemoveContainer" containerID="e53038da1d4b9cdd55a7f432efe241846fa21d27a70be20591bcbe9f9cf982d0" Oct 29 23:51:31.405574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:31.405543 5736 scope.go:122] "RemoveContainer" containerID="bcff04e056bebeaa66a8682240560a9a942efd9b450bd068158dd8f0793e3b51" Oct 29 23:51:31.416319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:31.416298 5736 scope.go:122] "RemoveContainer" containerID="6d1122b89cb54481c53f9935fa781d3397f892963e7fabca80c80006c8a4e566" Oct 29 23:51:32.469108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.469069 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz"] Oct 29 23:51:32.469575 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.469276 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" podUID="d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" containerName="maas-api" containerID="cri-o://7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061" gracePeriod=30 Oct 29 23:51:32.858867 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.858825 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:51:32.911571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.911535 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-openshift-service-ca\") pod \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " Oct 29 23:51:32.911798 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.911751 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-metrics-certs\") pod \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " Oct 29 23:51:32.911883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.911835 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-maas-api-tls\") pod \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " Oct 29 23:51:32.911949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.911901 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-kube-api-access-psvk6\" (UniqueName: \"kubernetes.io/projected/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-kube-api-access-psvk6\") pod \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\" (UID: \"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0\") " Oct 29 23:51:32.912006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.911959 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-openshift-service-ca" pod "d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" (UID: "d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:32.912491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.912464 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:32.914162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.914140 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-metrics-certs" pod "d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" (UID: "d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:32.914451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.914420 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-maas-api-tls" pod "d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" (UID: "d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:32.914796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:32.914769 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-kube-api-access-psvk6" pod "d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" (UID: "d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0"). InnerVolumeSpecName "kube-api-access-psvk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:33.021864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.015002 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:33.021864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.015041 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvk6\" (UniqueName: \"kubernetes.io/projected/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-kube-api-access-psvk6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:33.021864 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.015053 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:33.170120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.170038 5736 generic.go:356] "Generic (PLEG): container finished" podID="d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" containerID="7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061" exitCode=0 Oct 29 23:51:33.170120 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.170101 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" Oct 29 23:51:33.170376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.170126 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" event={"ID":"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0","Type":"ContainerDied","Data":"7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061"} Oct 29 23:51:33.170376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.170193 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz" event={"ID":"d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0","Type":"ContainerDied","Data":"b3c1b6c9cf02ecac09ac1468be1c052f1443385f96a2b3fb756e15e3b9ec8ec8"} Oct 29 23:51:33.170376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.170225 5736 scope.go:122] "RemoveContainer" containerID="7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061" Oct 29 23:51:33.181902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.181880 5736 scope.go:122] "RemoveContainer" containerID="7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061" Oct 29 23:51:33.182181 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:51:33.182157 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061\": container with ID starting with 7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061 not found: ID does not exist" containerID="7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061" Oct 29 23:51:33.182289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.182187 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061"} err="failed to get container status \"7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061\": rpc error: code = NotFound desc = could not find container \"7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061\": container with ID starting with 7b78d80e83c91a0f185bf3f246e608be5edc29119b251b5632922dad71839061 not found: ID does not exist" Oct 29 23:51:33.222252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.222224 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz"] Oct 29 23:51:33.234001 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.233974 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d20345-86864bdffc-ppxpz"] Oct 29 23:51:33.374329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.374239 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn"] Oct 29 23:51:33.378005 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:33.377975 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d20345-2qdvn"] Oct 29 23:51:35.006151 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:35.006117 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c56f0ec-5138-47cd-892a-29b8df87075c" path="/var/lib/kubelet/pods/3c56f0ec-5138-47cd-892a-29b8df87075c/volumes" Oct 29 23:51:35.006647 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:35.006625 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" path="/var/lib/kubelet/pods/d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0/volumes" Oct 29 23:51:35.193714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:35.192954 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr"] Oct 29 23:51:35.193714 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:35.193203 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" containerID="cri-o://84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af" gracePeriod=30 Oct 29 23:51:35.203191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:35.203165 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp"] Oct 29 23:51:35.203413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:35.203388 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" containerID="cri-o://c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b" gracePeriod=30 Oct 29 23:51:36.100887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.100851 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn"] Oct 29 23:51:36.101490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.101470 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" containerName="maas-api" Oct 29 23:51:36.101537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.101491 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" containerName="maas-api" Oct 29 23:51:36.101537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.101504 5736 cpu_manager.go:414] "containerMap: removing container" podUID="bef6171a-064f-4f9a-a65d-6fed44a00d36" containerName="revoke-keys" Oct 29 23:51:36.101537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.101510 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef6171a-064f-4f9a-a65d-6fed44a00d36" containerName="revoke-keys" Oct 29 23:51:36.101660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.101640 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b11d7a-4eac-42c3-ba4e-ab8aaf081bc0" containerName="maas-api" Oct 29 23:51:36.101660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.101649 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef6171a-064f-4f9a-a65d-6fed44a00d36" containerName="revoke-keys" Oct 29 23:51:36.106816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.106795 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.122731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.122710 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:51:36.134895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.134860 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:51:36.139284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.139249 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn"] Oct 29 23:51:36.168758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.168710 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfcln\" (UniqueName: \"kubernetes.io/projected/a2dbac38-12ca-4340-98b9-e677db183f0e-kube-api-access-jfcln\") pod \"maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.169341 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.169307 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.169457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.169404 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.270219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.270186 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.270357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.270254 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.270357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.270276 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfcln\" (UniqueName: \"kubernetes.io/projected/a2dbac38-12ca-4340-98b9-e677db183f0e-kube-api-access-jfcln\") pod \"maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.271010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.270981 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.272748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.272719 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.296259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.296228 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfcln\" (UniqueName: \"kubernetes.io/projected/a2dbac38-12ca-4340-98b9-e677db183f0e-kube-api-access-jfcln\") pod \"maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.421836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.421764 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:36.726101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:36.726036 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn"] Oct 29 23:51:36.728922 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:51:36.728879 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dbac38_12ca_4340_98b9_e677db183f0e.slice/crio-5ff2213f74f8710ad95b23865c94ce48782fda3f76c6d5f964a134674595b0b0 WatchSource:0}: Error finding container 5ff2213f74f8710ad95b23865c94ce48782fda3f76c6d5f964a134674595b0b0: Status 404 returned error can't find the container with id 5ff2213f74f8710ad95b23865c94ce48782fda3f76c6d5f964a134674595b0b0 Oct 29 23:51:37.216094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.216062 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" event={"ID":"a2dbac38-12ca-4340-98b9-e677db183f0e","Type":"ContainerStarted","Data":"19f0331b6c2703a0116652ec13b449d06779e96cd56b14f541a4b04aefd6fcb0"} Oct 29 23:51:37.216451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.216112 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" event={"ID":"a2dbac38-12ca-4340-98b9-e677db183f0e","Type":"ContainerStarted","Data":"5ff2213f74f8710ad95b23865c94ce48782fda3f76c6d5f964a134674595b0b0"} Oct 29 23:51:37.258924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.258879 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" podStartSLOduration=1.25886614 podStartE2EDuration="1.25886614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:51:37.255510764 +0000 UTC m=+4094.371430395" watchObservedRunningTime="2026-10-29 23:51:37.25886614 +0000 UTC m=+4094.374785746" Oct 29 23:51:37.504502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.504453 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" podUID="b131f7e8-db27-4412-9886-6556853efcba" containerName="main" containerID="cri-o://8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8" gracePeriod=45 Oct 29 23:51:37.765199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.765128 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:51:37.892351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.892312 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b131f7e8-db27-4412-9886-6556853efcba-tls-certs\" (UniqueName: \"kubernetes.io/secret/b131f7e8-db27-4412-9886-6556853efcba-tls-certs\") pod \"b131f7e8-db27-4412-9886-6556853efcba\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " Oct 29 23:51:37.892515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.892456 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-dshm\") pod \"b131f7e8-db27-4412-9886-6556853efcba\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " Oct 29 23:51:37.892557 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.892513 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-model-cache\") pod \"b131f7e8-db27-4412-9886-6556853efcba\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " Oct 29 23:51:37.892616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.892568 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b131f7e8-db27-4412-9886-6556853efcba-kube-api-access-b9txj\" (UniqueName: \"kubernetes.io/projected/b131f7e8-db27-4412-9886-6556853efcba-kube-api-access-b9txj\") pod \"b131f7e8-db27-4412-9886-6556853efcba\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " Oct 29 23:51:37.892616 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.892605 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-home\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-home\") pod \"b131f7e8-db27-4412-9886-6556853efcba\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " Oct 29 23:51:37.892756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.892734 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-tmp-dir\") pod \"b131f7e8-db27-4412-9886-6556853efcba\" (UID: \"b131f7e8-db27-4412-9886-6556853efcba\") " Oct 29 23:51:37.892855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.892833 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-model-cache" pod "b131f7e8-db27-4412-9886-6556853efcba" (UID: "b131f7e8-db27-4412-9886-6556853efcba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:37.892961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.892927 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-home" pod "b131f7e8-db27-4412-9886-6556853efcba" (UID: "b131f7e8-db27-4412-9886-6556853efcba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:37.893101 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.893057 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-tmp-dir" pod "b131f7e8-db27-4412-9886-6556853efcba" (UID: "b131f7e8-db27-4412-9886-6556853efcba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:37.893421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.893398 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:37.893477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.893427 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:37.893477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.893443 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:37.894762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.894743 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b131f7e8-db27-4412-9886-6556853efcba-tls-certs" pod "b131f7e8-db27-4412-9886-6556853efcba" (UID: "b131f7e8-db27-4412-9886-6556853efcba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:37.894832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.894764 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-dshm" pod "b131f7e8-db27-4412-9886-6556853efcba" (UID: "b131f7e8-db27-4412-9886-6556853efcba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:37.894877 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.894828 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b131f7e8-db27-4412-9886-6556853efcba-kube-api-access-b9txj" pod "b131f7e8-db27-4412-9886-6556853efcba" (UID: "b131f7e8-db27-4412-9886-6556853efcba"). InnerVolumeSpecName "kube-api-access-b9txj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:37.994789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.994757 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b131f7e8-db27-4412-9886-6556853efcba-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:37.994789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.994784 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9txj\" (UniqueName: \"kubernetes.io/projected/b131f7e8-db27-4412-9886-6556853efcba-kube-api-access-b9txj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:37.994789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:37.994795 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b131f7e8-db27-4412-9886-6556853efcba-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:38.224468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.224435 5736 generic.go:356] "Generic (PLEG): container finished" podID="b131f7e8-db27-4412-9886-6556853efcba" containerID="8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8" exitCode=0 Oct 29 23:51:38.224939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.224483 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" Oct 29 23:51:38.224939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.224483 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" event={"ID":"b131f7e8-db27-4412-9886-6556853efcba","Type":"ContainerDied","Data":"8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8"} Oct 29 23:51:38.224939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.224619 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49" event={"ID":"b131f7e8-db27-4412-9886-6556853efcba","Type":"ContainerDied","Data":"3f04bc10752c829c8c5f9266713bd91f5a2e56de93ee8fcad99c0f925bfa36b8"} Oct 29 23:51:38.224939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.224641 5736 scope.go:122] "RemoveContainer" containerID="8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8" Oct 29 23:51:38.226507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.226480 5736 generic.go:356] "Generic (PLEG): container finished" podID="a2dbac38-12ca-4340-98b9-e677db183f0e" containerID="19f0331b6c2703a0116652ec13b449d06779e96cd56b14f541a4b04aefd6fcb0" exitCode=0 Oct 29 23:51:38.226651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.226517 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" event={"ID":"a2dbac38-12ca-4340-98b9-e677db183f0e","Type":"ContainerDied","Data":"19f0331b6c2703a0116652ec13b449d06779e96cd56b14f541a4b04aefd6fcb0"} Oct 29 23:51:38.235658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.235638 5736 scope.go:122] "RemoveContainer" containerID="8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8" Oct 29 23:51:38.235934 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:51:38.235913 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8\": container with ID starting with 8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8 not found: ID does not exist" containerID="8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8" Oct 29 23:51:38.235997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.235941 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8"} err="failed to get container status \"8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8\": rpc error: code = NotFound desc = could not find container \"8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8\": container with ID starting with 8999a90720cfa834889897306d45ebf1c6a49fbceddd44ce46b38b28a8e06cb8 not found: ID does not exist" Oct 29 23:51:38.317253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.317223 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49"] Oct 29 23:51:38.320306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:38.320278 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-37498a/test-model-37498a-kserve-c58d954f5-hzm49"] Oct 29 23:51:39.004863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.004831 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b131f7e8-db27-4412-9886-6556853efcba" path="/var/lib/kubelet/pods/b131f7e8-db27-4412-9886-6556853efcba/volumes" Oct 29 23:51:39.405029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.405002 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:39.514874 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.514834 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-openshift-ca\") pod \"a2dbac38-12ca-4340-98b9-e677db183f0e\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " Oct 29 23:51:39.515079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.514902 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-xks-ca\") pod \"a2dbac38-12ca-4340-98b9-e677db183f0e\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " Oct 29 23:51:39.515079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.515040 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2dbac38-12ca-4340-98b9-e677db183f0e-kube-api-access-jfcln\" (UniqueName: \"kubernetes.io/projected/a2dbac38-12ca-4340-98b9-e677db183f0e-kube-api-access-jfcln\") pod \"a2dbac38-12ca-4340-98b9-e677db183f0e\" (UID: \"a2dbac38-12ca-4340-98b9-e677db183f0e\") " Oct 29 23:51:39.515355 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.515322 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-openshift-ca" pod "a2dbac38-12ca-4340-98b9-e677db183f0e" (UID: "a2dbac38-12ca-4340-98b9-e677db183f0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:39.515672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.515649 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:39.517812 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.517783 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dbac38-12ca-4340-98b9-e677db183f0e-kube-api-access-jfcln" pod "a2dbac38-12ca-4340-98b9-e677db183f0e" (UID: "a2dbac38-12ca-4340-98b9-e677db183f0e"). InnerVolumeSpecName "kube-api-access-jfcln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:39.517956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.517827 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-xks-ca" pod "a2dbac38-12ca-4340-98b9-e677db183f0e" (UID: "a2dbac38-12ca-4340-98b9-e677db183f0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:39.616568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.616507 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfcln\" (UniqueName: \"kubernetes.io/projected/a2dbac38-12ca-4340-98b9-e677db183f0e-kube-api-access-jfcln\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:39.616568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:39.616533 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dbac38-12ca-4340-98b9-e677db183f0e-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.252230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.252188 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" event={"ID":"a2dbac38-12ca-4340-98b9-e677db183f0e","Type":"ContainerDied","Data":"5ff2213f74f8710ad95b23865c94ce48782fda3f76c6d5f964a134674595b0b0"} Oct 29 23:51:40.252477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.252457 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff2213f74f8710ad95b23865c94ce48782fda3f76c6d5f964a134674595b0b0" Oct 29 23:51:40.252739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.252722 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn" Oct 29 23:51:40.269378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.269353 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w"] Oct 29 23:51:40.270113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.270091 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a2dbac38-12ca-4340-98b9-e677db183f0e" containerName="revoke-keys" Oct 29 23:51:40.270213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.270123 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2dbac38-12ca-4340-98b9-e677db183f0e" containerName="revoke-keys" Oct 29 23:51:40.270213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.270154 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b131f7e8-db27-4412-9886-6556853efcba" containerName="main" Oct 29 23:51:40.270213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.270159 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b131f7e8-db27-4412-9886-6556853efcba" containerName="main" Oct 29 23:51:40.270314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.270258 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2dbac38-12ca-4340-98b9-e677db183f0e" containerName="revoke-keys" Oct 29 23:51:40.270314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.270271 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b131f7e8-db27-4412-9886-6556853efcba" containerName="main" Oct 29 23:51:40.273785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.273767 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.296160 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.296112 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:51:40.296345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.296121 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:51:40.301024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.300997 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w"] Oct 29 23:51:40.337052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.336767 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb7b5\" (UniqueName: \"kubernetes.io/projected/b4e27c60-2a27-4b21-9f47-1572042e5b29-kube-api-access-xb7b5\") pod \"maas-api-revoke-keys-e2e-inf-a-37498a-67z8w\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.337052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.336851 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-37498a-67z8w\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.337052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.336951 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-37498a-67z8w\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.439100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.439066 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb7b5\" (UniqueName: \"kubernetes.io/projected/b4e27c60-2a27-4b21-9f47-1572042e5b29-kube-api-access-xb7b5\") pod \"maas-api-revoke-keys-e2e-inf-a-37498a-67z8w\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.439445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.439165 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-37498a-67z8w\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.439445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.439246 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-37498a-67z8w\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.440437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.440384 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-37498a-67z8w\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.443215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.443188 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-37498a-67z8w\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.462552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.462518 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb7b5\" (UniqueName: \"kubernetes.io/projected/b4e27c60-2a27-4b21-9f47-1572042e5b29-kube-api-access-xb7b5\") pod \"maas-api-revoke-keys-e2e-inf-a-37498a-67z8w\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.535080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.535051 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:51:40.571957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.571925 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:40.642502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642466 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-socket\") pod \"265571ba-40d8-4fa9-92dd-162d252f09a4\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " Oct 29 23:51:40.642722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642554 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265571ba-40d8-4fa9-92dd-162d252f09a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265571ba-40d8-4fa9-92dd-162d252f09a4-istiod-ca-cert\") pod \"265571ba-40d8-4fa9-92dd-162d252f09a4\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " Oct 29 23:51:40.642722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642612 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-kube-api-access-nhvcl\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-kube-api-access-nhvcl\") pod \"265571ba-40d8-4fa9-92dd-162d252f09a4\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " Oct 29 23:51:40.642722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642680 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-socket\") pod \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " Oct 29 23:51:40.642930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642750 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-data\") pod \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " Oct 29 23:51:40.642930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642791 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-certs\") pod \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " Oct 29 23:51:40.642930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642832 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-certs\") pod \"265571ba-40d8-4fa9-92dd-162d252f09a4\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " Oct 29 23:51:40.642930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642869 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-podinfo\") pod \"265571ba-40d8-4fa9-92dd-162d252f09a4\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " Oct 29 23:51:40.642930 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642915 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-envoy\") pod \"265571ba-40d8-4fa9-92dd-162d252f09a4\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " Oct 29 23:51:40.643246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642964 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-token\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-token\") pod \"265571ba-40d8-4fa9-92dd-162d252f09a4\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " Oct 29 23:51:40.643246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643006 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-token\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-token\") pod \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " Oct 29 23:51:40.643246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643124 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-kube-api-access-q67vm\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-kube-api-access-q67vm\") pod \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " Oct 29 23:51:40.643246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643177 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istiod-ca-cert\") pod \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " Oct 29 23:51:40.643246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643220 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-data\") pod \"265571ba-40d8-4fa9-92dd-162d252f09a4\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " Oct 29 23:51:40.643246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642789 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-socket" pod "265571ba-40d8-4fa9-92dd-162d252f09a4" (UID: "265571ba-40d8-4fa9-92dd-162d252f09a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.643630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643252 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-envoy\") pod \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " Oct 29 23:51:40.643630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.642979 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-socket" pod "efc17b25-421e-405d-ba4f-ebd1af1cdc0b" (UID: "efc17b25-421e-405d-ba4f-ebd1af1cdc0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.643630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643003 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265571ba-40d8-4fa9-92dd-162d252f09a4-istiod-ca-cert" pod "265571ba-40d8-4fa9-92dd-162d252f09a4" (UID: "265571ba-40d8-4fa9-92dd-162d252f09a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:40.643630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643228 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-certs" pod "265571ba-40d8-4fa9-92dd-162d252f09a4" (UID: "265571ba-40d8-4fa9-92dd-162d252f09a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.643630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643309 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-podinfo\") pod \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " Oct 29 23:51:40.643630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643339 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-credential-socket\") pod \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\" (UID: \"efc17b25-421e-405d-ba4f-ebd1af1cdc0b\") " Oct 29 23:51:40.643630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643376 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-credential-socket\") pod \"265571ba-40d8-4fa9-92dd-162d252f09a4\" (UID: \"265571ba-40d8-4fa9-92dd-162d252f09a4\") " Oct 29 23:51:40.643630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643490 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-certs" pod "efc17b25-421e-405d-ba4f-ebd1af1cdc0b" (UID: "efc17b25-421e-405d-ba4f-ebd1af1cdc0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643682 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-data" pod "efc17b25-421e-405d-ba4f-ebd1af1cdc0b" (UID: "efc17b25-421e-405d-ba4f-ebd1af1cdc0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.643862 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-credential-socket" pod "265571ba-40d8-4fa9-92dd-162d252f09a4" (UID: "265571ba-40d8-4fa9-92dd-162d252f09a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.644102 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.644121 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.644132 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265571ba-40d8-4fa9-92dd-162d252f09a4-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.644141 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.644150 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.644160 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.644168 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.644351 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-data" pod "265571ba-40d8-4fa9-92dd-162d252f09a4" (UID: "265571ba-40d8-4fa9-92dd-162d252f09a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.645229 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.644735 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istiod-ca-cert" pod "efc17b25-421e-405d-ba4f-ebd1af1cdc0b" (UID: "efc17b25-421e-405d-ba4f-ebd1af1cdc0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:40.646693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.646661 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-credential-socket" pod "efc17b25-421e-405d-ba4f-ebd1af1cdc0b" (UID: "efc17b25-421e-405d-ba4f-ebd1af1cdc0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.647058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.647028 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-podinfo" pod "efc17b25-421e-405d-ba4f-ebd1af1cdc0b" (UID: "efc17b25-421e-405d-ba4f-ebd1af1cdc0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:51:40.647982 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.647956 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-kube-api-access-nhvcl" pod "265571ba-40d8-4fa9-92dd-162d252f09a4" (UID: "265571ba-40d8-4fa9-92dd-162d252f09a4"). InnerVolumeSpecName "kube-api-access-nhvcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:40.650215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.650190 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-envoy" pod "efc17b25-421e-405d-ba4f-ebd1af1cdc0b" (UID: "efc17b25-421e-405d-ba4f-ebd1af1cdc0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.650215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.650197 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-podinfo" pod "265571ba-40d8-4fa9-92dd-162d252f09a4" (UID: "265571ba-40d8-4fa9-92dd-162d252f09a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:51:40.650386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.650212 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-kube-api-access-q67vm" pod "efc17b25-421e-405d-ba4f-ebd1af1cdc0b" (UID: "efc17b25-421e-405d-ba4f-ebd1af1cdc0b"). InnerVolumeSpecName "kube-api-access-q67vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:40.650386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.650225 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-token" pod "efc17b25-421e-405d-ba4f-ebd1af1cdc0b" (UID: "efc17b25-421e-405d-ba4f-ebd1af1cdc0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:40.650386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.650231 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-token" pod "265571ba-40d8-4fa9-92dd-162d252f09a4" (UID: "265571ba-40d8-4fa9-92dd-162d252f09a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:40.651381 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.651357 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-envoy" pod "265571ba-40d8-4fa9-92dd-162d252f09a4" (UID: "265571ba-40d8-4fa9-92dd-162d252f09a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:40.684705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.683748 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.763866 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.763904 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.763920 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q67vm\" (UniqueName: \"kubernetes.io/projected/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-kube-api-access-q67vm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.763936 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.763949 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.763961 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.763972 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.763984 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc17b25-421e-405d-ba4f-ebd1af1cdc0b-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.764000 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhvcl\" (UniqueName: \"kubernetes.io/projected/265571ba-40d8-4fa9-92dd-162d252f09a4-kube-api-access-nhvcl\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.764015 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.769277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.764029 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265571ba-40d8-4fa9-92dd-162d252f09a4-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:40.786325 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.772180 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p"] Oct 29 23:51:40.795641 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.791791 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm"] Oct 29 23:51:40.802403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:40.797468 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz"] Oct 29 23:51:41.150945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.150763 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w"] Oct 29 23:51:41.194128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.194095 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2"] Oct 29 23:51:41.194801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.194774 5736 cpu_manager.go:414] "containerMap: removing container" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" Oct 29 23:51:41.194801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.194801 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" Oct 29 23:51:41.194959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.194832 5736 cpu_manager.go:414] "containerMap: removing container" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" Oct 29 23:51:41.194959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.194840 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" Oct 29 23:51:41.195036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.195002 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerName="istio-proxy" Oct 29 23:51:41.195036 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.195019 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerName="istio-proxy" Oct 29 23:51:41.200250 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.200225 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.220413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.220386 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2"] Oct 29 23:51:41.263097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.263061 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" event={"ID":"b4e27c60-2a27-4b21-9f47-1572042e5b29","Type":"ContainerStarted","Data":"42ee0e979bff57e9f20becbc03787780b89c28974c0f067df013c0176ef7c0ce"} Oct 29 23:51:41.264734 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.264706 5736 generic.go:356] "Generic (PLEG): container finished" podID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" containerID="c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b" exitCode=0 Oct 29 23:51:41.264870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.264772 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" Oct 29 23:51:41.264870 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.264788 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" event={"ID":"efc17b25-421e-405d-ba4f-ebd1af1cdc0b","Type":"ContainerDied","Data":"c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b"} Oct 29 23:51:41.264978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.264877 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp" event={"ID":"efc17b25-421e-405d-ba4f-ebd1af1cdc0b","Type":"ContainerDied","Data":"4f69e333464158be7a17b1ddba0d18ce112eb9f122e711a911a7caa5e6bbf0cf"} Oct 29 23:51:41.264978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.264915 5736 scope.go:122] "RemoveContainer" containerID="c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b" Oct 29 23:51:41.267019 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.266990 5736 generic.go:356] "Generic (PLEG): container finished" podID="265571ba-40d8-4fa9-92dd-162d252f09a4" containerID="84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af" exitCode=0 Oct 29 23:51:41.267125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.267051 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" Oct 29 23:51:41.267207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.267134 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" event={"ID":"265571ba-40d8-4fa9-92dd-162d252f09a4","Type":"ContainerDied","Data":"84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af"} Oct 29 23:51:41.267207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.267197 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr" event={"ID":"265571ba-40d8-4fa9-92dd-162d252f09a4","Type":"ContainerDied","Data":"4b7f3b71ab3ae8ac4bb678b20e00c56d0284d492960dc788a7c0ebbb84b69ff9"} Oct 29 23:51:41.276772 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.276739 5736 scope.go:122] "RemoveContainer" containerID="c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b" Oct 29 23:51:41.277038 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:51:41.277017 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b\": container with ID starting with c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b not found: ID does not exist" containerID="c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b" Oct 29 23:51:41.277125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.277050 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b"} err="failed to get container status \"c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b\": rpc error: code = NotFound desc = could not find container \"c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b\": container with ID starting with c2b011120c1204fb76c5db56e70936fc5e6007751af8e083d64d0eb569141a3b not found: ID does not exist" Oct 29 23:51:41.277125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.277073 5736 scope.go:122] "RemoveContainer" containerID="84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af" Oct 29 23:51:41.286983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.286959 5736 scope.go:122] "RemoveContainer" containerID="84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af" Oct 29 23:51:41.287270 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:51:41.287249 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af\": container with ID starting with 84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af not found: ID does not exist" containerID="84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af" Oct 29 23:51:41.287336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.287281 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af"} err="failed to get container status \"84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af\": rpc error: code = NotFound desc = could not find container \"84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af\": container with ID starting with 84259c60edad063a9b3e1d4d3b6f743bb5bd620eeab36e698ec3a9812d5f23af not found: ID does not exist" Oct 29 23:51:41.317917 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.317883 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr"] Oct 29 23:51:41.329661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.329627 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-k5fxr"] Oct 29 23:51:41.355061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.355020 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp"] Oct 29 23:51:41.358792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.358761 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-n94sp"] Oct 29 23:51:41.370871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.370837 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.371006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.370913 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/37ebe08e-df79-4813-81a3-2709724d8e48-kube-api-access-8s4xk\") pod \"maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.371071 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.371049 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.472936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.472896 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.473322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.473129 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.473322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.473261 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/37ebe08e-df79-4813-81a3-2709724d8e48-kube-api-access-8s4xk\") pod \"maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.473728 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.473704 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.475703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.475678 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.497404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.497375 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/37ebe08e-df79-4813-81a3-2709724d8e48-kube-api-access-8s4xk\") pod \"maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.515275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.515219 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:41.830730 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:41.830700 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2"] Oct 29 23:51:41.833101 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:51:41.833064 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ebe08e_df79_4813_81a3_2709724d8e48.slice/crio-e60562c9399c92ff046f3174a54a021514036930ebedec1fe7d309920c36df50 WatchSource:0}: Error finding container e60562c9399c92ff046f3174a54a021514036930ebedec1fe7d309920c36df50: Status 404 returned error can't find the container with id e60562c9399c92ff046f3174a54a021514036930ebedec1fe7d309920c36df50 Oct 29 23:51:42.276889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:42.276827 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" event={"ID":"37ebe08e-df79-4813-81a3-2709724d8e48","Type":"ContainerStarted","Data":"5f1628232f74529f1b408efd6159783ee952ac1b766987a54d6f9aef823ee2ae"} Oct 29 23:51:42.277125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:42.276915 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" event={"ID":"37ebe08e-df79-4813-81a3-2709724d8e48","Type":"ContainerStarted","Data":"e60562c9399c92ff046f3174a54a021514036930ebedec1fe7d309920c36df50"} Oct 29 23:51:42.278716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:42.278688 5736 generic.go:356] "Generic (PLEG): container finished" podID="b4e27c60-2a27-4b21-9f47-1572042e5b29" containerID="73c4f24e90d982fb413e716d3c5a65f73b6778646c3dec0022bc22823792ef1d" exitCode=0 Oct 29 23:51:42.278862 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:42.278770 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" event={"ID":"b4e27c60-2a27-4b21-9f47-1572042e5b29","Type":"ContainerDied","Data":"73c4f24e90d982fb413e716d3c5a65f73b6778646c3dec0022bc22823792ef1d"} Oct 29 23:51:42.316894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:42.316845 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" podStartSLOduration=1.316831999 podStartE2EDuration="1.316831999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:51:42.315924953 +0000 UTC m=+4099.431844559" watchObservedRunningTime="2026-10-29 23:51:42.316831999 +0000 UTC m=+4099.432751642" Oct 29 23:51:43.003509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.003467 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265571ba-40d8-4fa9-92dd-162d252f09a4" path="/var/lib/kubelet/pods/265571ba-40d8-4fa9-92dd-162d252f09a4/volumes" Oct 29 23:51:43.004033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.004015 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc17b25-421e-405d-ba4f-ebd1af1cdc0b" path="/var/lib/kubelet/pods/efc17b25-421e-405d-ba4f-ebd1af1cdc0b/volumes" Oct 29 23:51:43.296808 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.296766 5736 generic.go:356] "Generic (PLEG): container finished" podID="37ebe08e-df79-4813-81a3-2709724d8e48" containerID="5f1628232f74529f1b408efd6159783ee952ac1b766987a54d6f9aef823ee2ae" exitCode=0 Oct 29 23:51:43.296992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.296956 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" event={"ID":"37ebe08e-df79-4813-81a3-2709724d8e48","Type":"ContainerDied","Data":"5f1628232f74529f1b408efd6159783ee952ac1b766987a54d6f9aef823ee2ae"} Oct 29 23:51:43.465570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.465541 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:43.600662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.600549 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-openshift-ca\") pod \"b4e27c60-2a27-4b21-9f47-1572042e5b29\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " Oct 29 23:51:43.600831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.600816 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e27c60-2a27-4b21-9f47-1572042e5b29-kube-api-access-xb7b5\" (UniqueName: \"kubernetes.io/projected/b4e27c60-2a27-4b21-9f47-1572042e5b29-kube-api-access-xb7b5\") pod \"b4e27c60-2a27-4b21-9f47-1572042e5b29\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " Oct 29 23:51:43.600904 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.600893 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-xks-ca\") pod \"b4e27c60-2a27-4b21-9f47-1572042e5b29\" (UID: \"b4e27c60-2a27-4b21-9f47-1572042e5b29\") " Oct 29 23:51:43.600946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.600913 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-openshift-ca" pod "b4e27c60-2a27-4b21-9f47-1572042e5b29" (UID: "b4e27c60-2a27-4b21-9f47-1572042e5b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:43.601233 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.601218 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:43.603007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.602984 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e27c60-2a27-4b21-9f47-1572042e5b29-kube-api-access-xb7b5" pod "b4e27c60-2a27-4b21-9f47-1572042e5b29" (UID: "b4e27c60-2a27-4b21-9f47-1572042e5b29"). InnerVolumeSpecName "kube-api-access-xb7b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:43.603150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.603081 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-xks-ca" pod "b4e27c60-2a27-4b21-9f47-1572042e5b29" (UID: "b4e27c60-2a27-4b21-9f47-1572042e5b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:43.702694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.702660 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb7b5\" (UniqueName: \"kubernetes.io/projected/b4e27c60-2a27-4b21-9f47-1572042e5b29-kube-api-access-xb7b5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:43.702694 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:43.702688 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e27c60-2a27-4b21-9f47-1572042e5b29-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:44.305144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.305111 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" Oct 29 23:51:44.305553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.305148 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w" event={"ID":"b4e27c60-2a27-4b21-9f47-1572042e5b29","Type":"ContainerDied","Data":"42ee0e979bff57e9f20becbc03787780b89c28974c0f067df013c0176ef7c0ce"} Oct 29 23:51:44.305553 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.305205 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ee0e979bff57e9f20becbc03787780b89c28974c0f067df013c0176ef7c0ce" Oct 29 23:51:44.472924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.472899 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:44.615357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.615274 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-xks-ca\") pod \"37ebe08e-df79-4813-81a3-2709724d8e48\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " Oct 29 23:51:44.615509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.615375 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ebe08e-df79-4813-81a3-2709724d8e48-kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/37ebe08e-df79-4813-81a3-2709724d8e48-kube-api-access-8s4xk\") pod \"37ebe08e-df79-4813-81a3-2709724d8e48\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " Oct 29 23:51:44.615559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.615511 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-openshift-ca\") pod \"37ebe08e-df79-4813-81a3-2709724d8e48\" (UID: \"37ebe08e-df79-4813-81a3-2709724d8e48\") " Oct 29 23:51:44.615908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.615883 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-openshift-ca" pod "37ebe08e-df79-4813-81a3-2709724d8e48" (UID: "37ebe08e-df79-4813-81a3-2709724d8e48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:44.617454 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.617426 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-xks-ca" pod "37ebe08e-df79-4813-81a3-2709724d8e48" (UID: "37ebe08e-df79-4813-81a3-2709724d8e48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:44.617554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.617522 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ebe08e-df79-4813-81a3-2709724d8e48-kube-api-access-8s4xk" pod "37ebe08e-df79-4813-81a3-2709724d8e48" (UID: "37ebe08e-df79-4813-81a3-2709724d8e48"). InnerVolumeSpecName "kube-api-access-8s4xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:44.717320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.717287 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:44.717320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.717316 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ebe08e-df79-4813-81a3-2709724d8e48-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:44.717486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:44.717326 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/37ebe08e-df79-4813-81a3-2709724d8e48-kube-api-access-8s4xk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:45.314751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:45.314669 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" Oct 29 23:51:45.314751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:45.314704 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2" event={"ID":"37ebe08e-df79-4813-81a3-2709724d8e48","Type":"ContainerDied","Data":"e60562c9399c92ff046f3174a54a021514036930ebedec1fe7d309920c36df50"} Oct 29 23:51:45.315179 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:45.314755 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e60562c9399c92ff046f3174a54a021514036930ebedec1fe7d309920c36df50" Oct 29 23:51:46.551791 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:46.551753 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l"] Oct 29 23:51:47.533410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:47.533372 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst"] Oct 29 23:51:47.544928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:47.544899 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b8149-5plst"] Oct 29 23:51:49.004409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:49.004361 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e6eb7e-a38e-4e4f-8104-d049839bbcab" path="/var/lib/kubelet/pods/60e6eb7e-a38e-4e4f-8104-d049839bbcab/volumes" Oct 29 23:51:51.552047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:51.552005 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" podUID="1632fbd9-04f1-4a5b-a40f-e778a40a3e55" containerName="payload-processing" containerID="cri-o://13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46" gracePeriod=25 Oct 29 23:51:54.420380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.420342 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j"] Oct 29 23:51:54.421534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.421476 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" containerID="cri-o://1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84" gracePeriod=30 Oct 29 23:51:54.917216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.917179 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg"] Oct 29 23:51:54.917822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.917805 5736 cpu_manager.go:414] "containerMap: removing container" podUID="37ebe08e-df79-4813-81a3-2709724d8e48" containerName="revoke-keys" Oct 29 23:51:54.917822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.917822 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ebe08e-df79-4813-81a3-2709724d8e48" containerName="revoke-keys" Oct 29 23:51:54.917953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.917839 5736 cpu_manager.go:414] "containerMap: removing container" podUID="b4e27c60-2a27-4b21-9f47-1572042e5b29" containerName="revoke-keys" Oct 29 23:51:54.917953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.917846 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e27c60-2a27-4b21-9f47-1572042e5b29" containerName="revoke-keys" Oct 29 23:51:54.917953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.917934 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ebe08e-df79-4813-81a3-2709724d8e48" containerName="revoke-keys" Oct 29 23:51:54.917953 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.917943 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e27c60-2a27-4b21-9f47-1572042e5b29" containerName="revoke-keys" Oct 29 23:51:54.922702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.922681 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:54.942457 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.942424 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg"] Oct 29 23:51:54.967275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.967241 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:51:54.967421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:54.967246 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:51:55.000559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.000531 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-825472-2hcbg\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.000559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.000562 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlth7\" (UniqueName: \"kubernetes.io/projected/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-kube-api-access-jlth7\") pod \"maas-api-revoke-keys-e2e-inf-b-825472-2hcbg\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.000787 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.000707 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-825472-2hcbg\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.101693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.101651 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-825472-2hcbg\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.101693 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.101686 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlth7\" (UniqueName: \"kubernetes.io/projected/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-kube-api-access-jlth7\") pod \"maas-api-revoke-keys-e2e-inf-b-825472-2hcbg\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.101946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.101723 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-825472-2hcbg\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.102324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.102298 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-825472-2hcbg\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.104053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.104033 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-825472-2hcbg\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.125062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.125036 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlth7\" (UniqueName: \"kubernetes.io/projected/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-kube-api-access-jlth7\") pod \"maas-api-revoke-keys-e2e-inf-b-825472-2hcbg\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.236568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.236536 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:55.543740 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.543713 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg"] Oct 29 23:51:55.546311 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:51:55.546280 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c45f48_4dad_4c2c_aba1_e8f799a81dc7.slice/crio-6856d66c994d83857f757b933f321527b4984044173e61db18d23608db69b9dc WatchSource:0}: Error finding container 6856d66c994d83857f757b933f321527b4984044173e61db18d23608db69b9dc: Status 404 returned error can't find the container with id 6856d66c994d83857f757b933f321527b4984044173e61db18d23608db69b9dc Oct 29 23:51:55.833116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.833077 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" containerID="cri-o://46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e" gracePeriod=45 Oct 29 23:51:55.878781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.878721 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" containerID="cri-o://4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a" gracePeriod=45 Oct 29 23:51:55.892083 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:55.892028 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" containerID="cri-o://fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc" gracePeriod=45 Oct 29 23:51:56.196219 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.196194 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 23:51:56.200869 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.200845 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:51:56.231145 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.231124 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 23:51:56.325265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325183 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1aa2b1-192a-485e-9791-a71051fe0172-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1aa2b1-192a-485e-9791-a71051fe0172-tls-certs\") pod \"2a1aa2b1-192a-485e-9791-a71051fe0172\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " Oct 29 23:51:56.325265 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325250 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-kube-api-access-wflt5\" (UniqueName: \"kubernetes.io/projected/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-kube-api-access-wflt5\") pod \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " Oct 29 23:51:56.325492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325308 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-home\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-home\") pod \"2a1aa2b1-192a-485e-9791-a71051fe0172\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " Oct 29 23:51:56.325492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325368 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a5318b-2f1c-4171-8242-2a9c94c517e4-kube-api-access-xj6np\" (UniqueName: \"kubernetes.io/projected/e6a5318b-2f1c-4171-8242-2a9c94c517e4-kube-api-access-xj6np\") pod \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " Oct 29 23:51:56.325492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325399 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1aa2b1-192a-485e-9791-a71051fe0172-kube-api-access-x7txz\" (UniqueName: \"kubernetes.io/projected/2a1aa2b1-192a-485e-9791-a71051fe0172-kube-api-access-x7txz\") pod \"2a1aa2b1-192a-485e-9791-a71051fe0172\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " Oct 29 23:51:56.325492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325452 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tmp-dir\") pod \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " Oct 29 23:51:56.325492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325482 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-dshm\") pod \"2a1aa2b1-192a-485e-9791-a71051fe0172\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " Oct 29 23:51:56.325884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325538 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tls-certs\") pod \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " Oct 29 23:51:56.325884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325573 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-model-cache\") pod \"2a1aa2b1-192a-485e-9791-a71051fe0172\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " Oct 29 23:51:56.325884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325635 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tls-certs\") pod \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " Oct 29 23:51:56.325884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325708 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-dshm\") pod \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " Oct 29 23:51:56.325884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325803 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-model-cache\") pod \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " Oct 29 23:51:56.325884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325838 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-model-cache\") pod \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " Oct 29 23:51:56.326201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325891 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-tmp-dir\") pod \"2a1aa2b1-192a-485e-9791-a71051fe0172\" (UID: \"2a1aa2b1-192a-485e-9791-a71051fe0172\") " Oct 29 23:51:56.326201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325932 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-dshm\") pod \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " Oct 29 23:51:56.326201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.325977 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-home\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-home\") pod \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\" (UID: \"e6a5318b-2f1c-4171-8242-2a9c94c517e4\") " Oct 29 23:51:56.326201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.326014 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tmp-dir\") pod \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " Oct 29 23:51:56.326201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.326045 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-home\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-home\") pod \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\" (UID: \"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4\") " Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.326920 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-home" pod "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" (UID: "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.327128 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-home" pod "e6a5318b-2f1c-4171-8242-2a9c94c517e4" (UID: "e6a5318b-2f1c-4171-8242-2a9c94c517e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.327137 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tmp-dir" pod "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" (UID: "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.327327 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-tmp-dir" pod "2a1aa2b1-192a-485e-9791-a71051fe0172" (UID: "2a1aa2b1-192a-485e-9791-a71051fe0172"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.327333 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-model-cache" pod "e6a5318b-2f1c-4171-8242-2a9c94c517e4" (UID: "e6a5318b-2f1c-4171-8242-2a9c94c517e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.327535 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-home" pod "2a1aa2b1-192a-485e-9791-a71051fe0172" (UID: "2a1aa2b1-192a-485e-9791-a71051fe0172"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.327869 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a5318b-2f1c-4171-8242-2a9c94c517e4-kube-api-access-xj6np" pod "e6a5318b-2f1c-4171-8242-2a9c94c517e4" (UID: "e6a5318b-2f1c-4171-8242-2a9c94c517e4"). InnerVolumeSpecName "kube-api-access-xj6np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.327909 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-model-cache" pod "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" (UID: "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.327928 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1aa2b1-192a-485e-9791-a71051fe0172-tls-certs" pod "2a1aa2b1-192a-485e-9791-a71051fe0172" (UID: "2a1aa2b1-192a-485e-9791-a71051fe0172"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.328228 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-model-cache" pod "2a1aa2b1-192a-485e-9791-a71051fe0172" (UID: "2a1aa2b1-192a-485e-9791-a71051fe0172"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.328465 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tmp-dir" pod "e6a5318b-2f1c-4171-8242-2a9c94c517e4" (UID: "e6a5318b-2f1c-4171-8242-2a9c94c517e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.331359 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1aa2b1-192a-485e-9791-a71051fe0172-kube-api-access-x7txz" pod "2a1aa2b1-192a-485e-9791-a71051fe0172" (UID: "2a1aa2b1-192a-485e-9791-a71051fe0172"). InnerVolumeSpecName "kube-api-access-x7txz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.331557 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tls-certs" pod "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" (UID: "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.331832 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tls-certs" pod "e6a5318b-2f1c-4171-8242-2a9c94c517e4" (UID: "e6a5318b-2f1c-4171-8242-2a9c94c517e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.332176 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-dshm" pod "2a1aa2b1-192a-485e-9791-a71051fe0172" (UID: "2a1aa2b1-192a-485e-9791-a71051fe0172"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.332247 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-kube-api-access-wflt5" pod "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" (UID: "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4"). InnerVolumeSpecName "kube-api-access-wflt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:56.337143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.334331 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-dshm" pod "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" (UID: "87b0f10e-ea68-4a85-a6d9-01a277a4b5b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.343711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.343649 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-dshm" pod "e6a5318b-2f1c-4171-8242-2a9c94c517e4" (UID: "e6a5318b-2f1c-4171-8242-2a9c94c517e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:56.427639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427605 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1aa2b1-192a-485e-9791-a71051fe0172-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427640 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wflt5\" (UniqueName: \"kubernetes.io/projected/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-kube-api-access-wflt5\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427656 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427674 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj6np\" (UniqueName: \"kubernetes.io/projected/e6a5318b-2f1c-4171-8242-2a9c94c517e4-kube-api-access-xj6np\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427686 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7txz\" (UniqueName: \"kubernetes.io/projected/2a1aa2b1-192a-485e-9791-a71051fe0172-kube-api-access-x7txz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427701 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427711 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427719 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a5318b-2f1c-4171-8242-2a9c94c517e4-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427729 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427741 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427753 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427764 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427779 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427791 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1aa2b1-192a-485e-9791-a71051fe0172-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427804 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427815 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a5318b-2f1c-4171-8242-2a9c94c517e4-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427827 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.427879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.427838 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:56.428574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.428544 5736 generic.go:356] "Generic (PLEG): container finished" podID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerID="fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc" exitCode=0 Oct 29 23:51:56.428677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.428611 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" event={"ID":"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4","Type":"ContainerDied","Data":"fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc"} Oct 29 23:51:56.428677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.428660 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" event={"ID":"87b0f10e-ea68-4a85-a6d9-01a277a4b5b4","Type":"ContainerDied","Data":"295cb1fed0ecf1901f8781916bd3823eaaef59c9499dee6c50b2bd9be73446cd"} Oct 29 23:51:56.428759 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.428623 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz" Oct 29 23:51:56.428796 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.428680 5736 scope.go:122] "RemoveContainer" containerID="fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc" Oct 29 23:51:56.430349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.430324 5736 generic.go:356] "Generic (PLEG): container finished" podID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerID="46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e" exitCode=0 Oct 29 23:51:56.430463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.430374 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" Oct 29 23:51:56.430463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.430449 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" event={"ID":"2a1aa2b1-192a-485e-9791-a71051fe0172","Type":"ContainerDied","Data":"46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e"} Oct 29 23:51:56.430561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.430494 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm" event={"ID":"2a1aa2b1-192a-485e-9791-a71051fe0172","Type":"ContainerDied","Data":"881e81d7934ba11b71cdeb2b983ba9451d9513d0c64f2e01838902c10bac7695"} Oct 29 23:51:56.431979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.431955 5736 generic.go:356] "Generic (PLEG): container finished" podID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerID="4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a" exitCode=0 Oct 29 23:51:56.432102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.432008 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" Oct 29 23:51:56.432102 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.432004 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" event={"ID":"e6a5318b-2f1c-4171-8242-2a9c94c517e4","Type":"ContainerDied","Data":"4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a"} Oct 29 23:51:56.432204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.432110 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p" event={"ID":"e6a5318b-2f1c-4171-8242-2a9c94c517e4","Type":"ContainerDied","Data":"c95d327d5cb866f381b5c29238070596999e1ac091c105933c42ae18c6a3237b"} Oct 29 23:51:56.438259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.438232 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" event={"ID":"08c45f48-4dad-4c2c-aba1-e8f799a81dc7","Type":"ContainerStarted","Data":"f0eb7e26219428d9d636e5137d602bbd0ba6c28dc4bd9a416071091dcb7813ff"} Oct 29 23:51:56.438380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.438283 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" event={"ID":"08c45f48-4dad-4c2c-aba1-e8f799a81dc7","Type":"ContainerStarted","Data":"6856d66c994d83857f757b933f321527b4984044173e61db18d23608db69b9dc"} Oct 29 23:51:56.440570 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.440553 5736 scope.go:122] "RemoveContainer" containerID="fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc" Oct 29 23:51:56.440878 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:51:56.440857 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc\": container with ID starting with fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc not found: ID does not exist" containerID="fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc" Oct 29 23:51:56.440956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.440884 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc"} err="failed to get container status \"fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc\": rpc error: code = NotFound desc = could not find container \"fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc\": container with ID starting with fb61be2f1149dabd620c136b16bf1f6636ae56c53e415c280944f08ae66a03dc not found: ID does not exist" Oct 29 23:51:56.440956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.440900 5736 scope.go:122] "RemoveContainer" containerID="46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e" Oct 29 23:51:56.451383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.451367 5736 scope.go:122] "RemoveContainer" containerID="46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e" Oct 29 23:51:56.451651 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:51:56.451632 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e\": container with ID starting with 46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e not found: ID does not exist" containerID="46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e" Oct 29 23:51:56.451705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.451657 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e"} err="failed to get container status \"46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e\": rpc error: code = NotFound desc = could not find container \"46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e\": container with ID starting with 46badc0df89ea759038513422b8007d74afa7ad64f0193ac916a0ebcadc5af3e not found: ID does not exist" Oct 29 23:51:56.451705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.451676 5736 scope.go:122] "RemoveContainer" containerID="4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a" Oct 29 23:51:56.461710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.461690 5736 scope.go:122] "RemoveContainer" containerID="4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a" Oct 29 23:51:56.462003 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:51:56.461983 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a\": container with ID starting with 4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a not found: ID does not exist" containerID="4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a" Oct 29 23:51:56.462054 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.462013 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a"} err="failed to get container status \"4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a\": rpc error: code = NotFound desc = could not find container \"4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a\": container with ID starting with 4bcd8ea2ebac9d67d847027a008cb9102088d3ea06fb698eed0bd612335a8e3a not found: ID does not exist" Oct 29 23:51:56.499308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.499252 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" podStartSLOduration=2.499233512 podStartE2EDuration="2.499233512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:51:56.498654504 +0000 UTC m=+4113.614574106" watchObservedRunningTime="2026-10-29 23:51:56.499233512 +0000 UTC m=+4113.615153122" Oct 29 23:51:56.551631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.551594 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz"] Oct 29 23:51:56.555201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.555176 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/simulator-w6-1ee571-kserve-5db5446bcc-xh9sz"] Oct 29 23:51:56.581446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.581416 5736 patch_prober.go:29] interesting pod/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:51:56.581624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.581468 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:51:56.588184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.588159 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p"] Oct 29 23:51:56.593675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.593651 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/distinct-w6-1ee571-kserve-55bbf744c9-b6b6p"] Oct 29 23:51:56.627315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.627248 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm"] Oct 29 23:51:56.631639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:56.631617 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1ee571/premium-w6-1ee571-kserve-846f9b65f6-cxpdm"] Oct 29 23:51:56.837886 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:51:56.829017 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:51:57.012292 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:57.012255 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" path="/var/lib/kubelet/pods/2a1aa2b1-192a-485e-9791-a71051fe0172/volumes" Oct 29 23:51:57.012783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:57.012765 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" path="/var/lib/kubelet/pods/87b0f10e-ea68-4a85-a6d9-01a277a4b5b4/volumes" Oct 29 23:51:57.013244 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:57.013227 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" path="/var/lib/kubelet/pods/e6a5318b-2f1c-4171-8242-2a9c94c517e4/volumes" Oct 29 23:51:57.452421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:57.452286 5736 generic.go:356] "Generic (PLEG): container finished" podID="08c45f48-4dad-4c2c-aba1-e8f799a81dc7" containerID="f0eb7e26219428d9d636e5137d602bbd0ba6c28dc4bd9a416071091dcb7813ff" exitCode=0 Oct 29 23:51:57.452421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:57.452340 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" event={"ID":"08c45f48-4dad-4c2c-aba1-e8f799a81dc7","Type":"ContainerDied","Data":"f0eb7e26219428d9d636e5137d602bbd0ba6c28dc4bd9a416071091dcb7813ff"} Oct 29 23:51:57.819005 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:57.818966 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2"] Oct 29 23:51:57.822201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:57.822170 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e1769a-vztf2"] Oct 29 23:51:58.583524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.583485 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" podUID="1632fbd9-04f1-4a5b-a40f-e778a40a3e55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:51:58.655144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.655115 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:58.681281 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.681250 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-openshift-ca\") pod \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " Oct 29 23:51:58.681429 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.681388 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-xks-ca\") pod \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " Oct 29 23:51:58.681592 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.681566 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-kube-api-access-jlth7\" (UniqueName: \"kubernetes.io/projected/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-kube-api-access-jlth7\") pod \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\" (UID: \"08c45f48-4dad-4c2c-aba1-e8f799a81dc7\") " Oct 29 23:51:58.681678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.681637 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-openshift-ca" pod "08c45f48-4dad-4c2c-aba1-e8f799a81dc7" (UID: "08c45f48-4dad-4c2c-aba1-e8f799a81dc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:58.682009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.681989 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:58.683471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.683449 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-kube-api-access-jlth7" pod "08c45f48-4dad-4c2c-aba1-e8f799a81dc7" (UID: "08c45f48-4dad-4c2c-aba1-e8f799a81dc7"). InnerVolumeSpecName "kube-api-access-jlth7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:58.683554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.683452 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-xks-ca" pod "08c45f48-4dad-4c2c-aba1-e8f799a81dc7" (UID: "08c45f48-4dad-4c2c-aba1-e8f799a81dc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:51:58.783458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.783356 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlth7\" (UniqueName: \"kubernetes.io/projected/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-kube-api-access-jlth7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:58.783458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:58.783400 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c45f48-4dad-4c2c-aba1-e8f799a81dc7-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.004154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.004112 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ebe08e-df79-4813-81a3-2709724d8e48" path="/var/lib/kubelet/pods/37ebe08e-df79-4813-81a3-2709724d8e48/volumes" Oct 29 23:51:59.503447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.503416 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" Oct 29 23:51:59.503447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.503434 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg" event={"ID":"08c45f48-4dad-4c2c-aba1-e8f799a81dc7","Type":"ContainerDied","Data":"6856d66c994d83857f757b933f321527b4984044173e61db18d23608db69b9dc"} Oct 29 23:51:59.503628 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.503485 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6856d66c994d83857f757b933f321527b4984044173e61db18d23608db69b9dc" Oct 29 23:51:59.687060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.687031 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:51:59.695212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695158 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc2652b-8879-4e8c-8337-ec8faa807dd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc2652b-8879-4e8c-8337-ec8faa807dd7-istiod-ca-cert\") pod \"afc2652b-8879-4e8c-8337-ec8faa807dd7\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " Oct 29 23:51:59.695212 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695194 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-token\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-token\") pod \"afc2652b-8879-4e8c-8337-ec8faa807dd7\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " Oct 29 23:51:59.695407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695221 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-credential-socket\") pod \"afc2652b-8879-4e8c-8337-ec8faa807dd7\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " Oct 29 23:51:59.695407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695273 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-data\") pod \"afc2652b-8879-4e8c-8337-ec8faa807dd7\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " Oct 29 23:51:59.695407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695313 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-socket\") pod \"afc2652b-8879-4e8c-8337-ec8faa807dd7\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " Oct 29 23:51:59.695407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695337 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-envoy\") pod \"afc2652b-8879-4e8c-8337-ec8faa807dd7\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " Oct 29 23:51:59.695407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695385 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-certs\") pod \"afc2652b-8879-4e8c-8337-ec8faa807dd7\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " Oct 29 23:51:59.695717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695409 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-podinfo\") pod \"afc2652b-8879-4e8c-8337-ec8faa807dd7\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " Oct 29 23:51:59.695717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695438 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-kube-api-access-6kdhv\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-kube-api-access-6kdhv\") pod \"afc2652b-8879-4e8c-8337-ec8faa807dd7\" (UID: \"afc2652b-8879-4e8c-8337-ec8faa807dd7\") " Oct 29 23:51:59.695717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695605 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-credential-socket" pod "afc2652b-8879-4e8c-8337-ec8faa807dd7" (UID: "afc2652b-8879-4e8c-8337-ec8faa807dd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.695717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695650 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-data" pod "afc2652b-8879-4e8c-8337-ec8faa807dd7" (UID: "afc2652b-8879-4e8c-8337-ec8faa807dd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.695717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695662 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-socket" pod "afc2652b-8879-4e8c-8337-ec8faa807dd7" (UID: "afc2652b-8879-4e8c-8337-ec8faa807dd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.696011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695727 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc2652b-8879-4e8c-8337-ec8faa807dd7-istiod-ca-cert" pod "afc2652b-8879-4e8c-8337-ec8faa807dd7" (UID: "afc2652b-8879-4e8c-8337-ec8faa807dd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:51:59.696011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695976 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-certs" pod "afc2652b-8879-4e8c-8337-ec8faa807dd7" (UID: "afc2652b-8879-4e8c-8337-ec8faa807dd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.696011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.695975 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc2652b-8879-4e8c-8337-ec8faa807dd7-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.696011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.696010 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.696246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.696021 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.696246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.696032 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.697519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.697491 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-token" pod "afc2652b-8879-4e8c-8337-ec8faa807dd7" (UID: "afc2652b-8879-4e8c-8337-ec8faa807dd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:59.697703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.697671 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-kube-api-access-6kdhv" pod "afc2652b-8879-4e8c-8337-ec8faa807dd7" (UID: "afc2652b-8879-4e8c-8337-ec8faa807dd7"). InnerVolumeSpecName "kube-api-access-6kdhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:51:59.698044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.698018 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-podinfo" pod "afc2652b-8879-4e8c-8337-ec8faa807dd7" (UID: "afc2652b-8879-4e8c-8337-ec8faa807dd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:51:59.698044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.698028 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-envoy" pod "afc2652b-8879-4e8c-8337-ec8faa807dd7" (UID: "afc2652b-8879-4e8c-8337-ec8faa807dd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:51:59.797445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.797410 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.797445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.797439 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.797445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.797448 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kdhv\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-kube-api-access-6kdhv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.797695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.797459 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:51:59.797695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:51:59.797469 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc2652b-8879-4e8c-8337-ec8faa807dd7-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:00.512486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.512448 5736 generic.go:356] "Generic (PLEG): container finished" podID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerID="1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84" exitCode=0 Oct 29 23:52:00.512961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.512505 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" Oct 29 23:52:00.512961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.512537 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" event={"ID":"afc2652b-8879-4e8c-8337-ec8faa807dd7","Type":"ContainerDied","Data":"1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84"} Oct 29 23:52:00.512961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.512624 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j" event={"ID":"afc2652b-8879-4e8c-8337-ec8faa807dd7","Type":"ContainerDied","Data":"af561e55a0af2480a58916bbdc6e9b332dcfbe384aa3cb4c1c802087aa5b4963"} Oct 29 23:52:00.512961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.512648 5736 scope.go:122] "RemoveContainer" containerID="1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84" Oct 29 23:52:00.524792 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.524772 5736 scope.go:122] "RemoveContainer" containerID="ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb" Oct 29 23:52:00.535308 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.535287 5736 scope.go:122] "RemoveContainer" containerID="1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84" Oct 29 23:52:00.535555 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:00.535538 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84\": container with ID starting with 1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84 not found: ID does not exist" containerID="1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84" Oct 29 23:52:00.535620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.535562 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84"} err="failed to get container status \"1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84\": rpc error: code = NotFound desc = could not find container \"1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84\": container with ID starting with 1dce97c9a5555e7096931f3c179cb33c3b7dc6db0646469703f32e8069634a84 not found: ID does not exist" Oct 29 23:52:00.535620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.535592 5736 scope.go:122] "RemoveContainer" containerID="ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb" Oct 29 23:52:00.535812 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:00.535794 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb\": container with ID starting with ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb not found: ID does not exist" containerID="ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb" Oct 29 23:52:00.535855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.535816 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb"} err="failed to get container status \"ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb\": rpc error: code = NotFound desc = could not find container \"ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb\": container with ID starting with ad679e03f933f4ea105d6220fd619da47c6cb781d4727044f9a81c3d6552e2fb not found: ID does not exist" Oct 29 23:52:00.571456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.571432 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j"] Oct 29 23:52:00.574246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:00.574219 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8149-openshift-default-bc984dd49-5g62j"] Oct 29 23:52:01.004222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:01.004189 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" path="/var/lib/kubelet/pods/afc2652b-8879-4e8c-8337-ec8faa807dd7/volumes" Oct 29 23:52:05.310732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:05.310691 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp"] Oct 29 23:52:08.583522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:08.583479 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" podUID="1632fbd9-04f1-4a5b-a40f-e778a40a3e55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:10.311851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.311806 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" podUID="82fed7e2-90a5-41e2-a1e0-22c694c3ef8b" containerName="payload-processing" containerID="cri-o://f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450" gracePeriod=25 Oct 29 23:52:10.440978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.440947 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d"] Oct 29 23:52:10.441613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441562 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" Oct 29 23:52:10.441613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441595 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" Oct 29 23:52:10.441613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441615 5736 cpu_manager.go:414] "containerMap: removing container" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" Oct 29 23:52:10.441839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441626 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" Oct 29 23:52:10.441839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441669 5736 cpu_manager.go:414] "containerMap: removing container" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" Oct 29 23:52:10.441839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441677 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" Oct 29 23:52:10.441839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441697 5736 cpu_manager.go:414] "containerMap: removing container" podUID="08c45f48-4dad-4c2c-aba1-e8f799a81dc7" containerName="revoke-keys" Oct 29 23:52:10.441839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441704 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c45f48-4dad-4c2c-aba1-e8f799a81dc7" containerName="revoke-keys" Oct 29 23:52:10.441839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441734 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" Oct 29 23:52:10.441839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441743 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" Oct 29 23:52:10.442230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441902 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1aa2b1-192a-485e-9791-a71051fe0172" containerName="main" Oct 29 23:52:10.442230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441916 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" Oct 29 23:52:10.442230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441932 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a5318b-2f1c-4171-8242-2a9c94c517e4" containerName="main" Oct 29 23:52:10.442230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441945 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" Oct 29 23:52:10.442230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441962 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b0f10e-ea68-4a85-a6d9-01a277a4b5b4" containerName="main" Oct 29 23:52:10.442230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.441972 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c45f48-4dad-4c2c-aba1-e8f799a81dc7" containerName="revoke-keys" Oct 29 23:52:10.445497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.445474 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.463032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.463003 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr"] Oct 29 23:52:10.463531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.463517 5736 cpu_manager.go:414] "containerMap: removing container" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" Oct 29 23:52:10.463599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.463532 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc2652b-8879-4e8c-8337-ec8faa807dd7" containerName="istio-proxy" Oct 29 23:52:10.467046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.467026 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.472623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.471318 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:52:10.472623 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.471930 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:52:10.480799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.478876 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d"] Oct 29 23:52:10.490134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.490106 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr"] Oct 29 23:52:10.557253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.557219 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d903d6-rs5dr\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.557404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.557329 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8lnk\" (UniqueName: \"kubernetes.io/projected/e8fb7bea-c53a-4cef-9400-c87c06b54554-kube-api-access-k8lnk\") pod \"maas-api-revoke-sub-ea0a4246-f8p8d\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.557463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.557444 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea0a4246-f8p8d\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.557503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.557484 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d903d6-rs5dr\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.557603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.557510 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea0a4246-f8p8d\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.557729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.557707 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpcmq\" (UniqueName: \"kubernetes.io/projected/2c2959a1-c03a-42aa-999e-135769ce1efc-kube-api-access-cpcmq\") pod \"maas-api-revoke-sub-60d903d6-rs5dr\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.600769 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.600740 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:52:10.620227 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.620192 5736 generic.go:356] "Generic (PLEG): container finished" podID="82fed7e2-90a5-41e2-a1e0-22c694c3ef8b" containerID="f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450" exitCode=0 Oct 29 23:52:10.620369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.620236 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" Oct 29 23:52:10.620369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.620258 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" event={"ID":"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b","Type":"ContainerDied","Data":"f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450"} Oct 29 23:52:10.620369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.620302 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp" event={"ID":"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b","Type":"ContainerDied","Data":"70dfc01ad4f0224433aac8b375f54e88c03bdc5de5ef2764b2a784c99b55318f"} Oct 29 23:52:10.620369 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.620323 5736 scope.go:122] "RemoveContainer" containerID="f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450" Oct 29 23:52:10.630871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.630854 5736 scope.go:122] "RemoveContainer" containerID="f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450" Oct 29 23:52:10.631135 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:10.631115 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450\": container with ID starting with f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450 not found: ID does not exist" containerID="f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450" Oct 29 23:52:10.631205 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.631147 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450"} err="failed to get container status \"f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450\": rpc error: code = NotFound desc = could not find container \"f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450\": container with ID starting with f73990eba60311156149a099aac9024cb3a8de4328a86988050369a2083aa450 not found: ID does not exist" Oct 29 23:52:10.658837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.658806 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpcmq\" (UniqueName: \"kubernetes.io/projected/2c2959a1-c03a-42aa-999e-135769ce1efc-kube-api-access-cpcmq\") pod \"maas-api-revoke-sub-60d903d6-rs5dr\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.659006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.658858 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d903d6-rs5dr\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.659006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.658903 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lnk\" (UniqueName: \"kubernetes.io/projected/e8fb7bea-c53a-4cef-9400-c87c06b54554-kube-api-access-k8lnk\") pod \"maas-api-revoke-sub-ea0a4246-f8p8d\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.659006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.658963 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea0a4246-f8p8d\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.659006 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.658992 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d903d6-rs5dr\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.659240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.659018 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea0a4246-f8p8d\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.659777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.659747 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea0a4246-f8p8d\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.659911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.659884 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d903d6-rs5dr\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.661719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.661693 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea0a4246-f8p8d\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.661821 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.661794 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d903d6-rs5dr\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.683021 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.682983 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpcmq\" (UniqueName: \"kubernetes.io/projected/2c2959a1-c03a-42aa-999e-135769ce1efc-kube-api-access-cpcmq\") pod \"maas-api-revoke-sub-60d903d6-rs5dr\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.683406 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.683388 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lnk\" (UniqueName: \"kubernetes.io/projected/e8fb7bea-c53a-4cef-9400-c87c06b54554-kube-api-access-k8lnk\") pod \"maas-api-revoke-sub-ea0a4246-f8p8d\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.759651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.759613 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-plugins-config-volume\") pod \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\" (UID: \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\") " Oct 29 23:52:10.759858 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.759778 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:10.759945 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.759888 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-kube-api-access-trccd\" (UniqueName: \"kubernetes.io/projected/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-kube-api-access-trccd\") pod \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\" (UID: \"82fed7e2-90a5-41e2-a1e0-22c694c3ef8b\") " Oct 29 23:52:10.760075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.760051 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-plugins-config-volume" pod "82fed7e2-90a5-41e2-a1e0-22c694c3ef8b" (UID: "82fed7e2-90a5-41e2-a1e0-22c694c3ef8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:10.760372 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.760348 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:10.762084 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.762055 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-kube-api-access-trccd" pod "82fed7e2-90a5-41e2-a1e0-22c694c3ef8b" (UID: "82fed7e2-90a5-41e2-a1e0-22c694c3ef8b"). InnerVolumeSpecName "kube-api-access-trccd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:10.786893 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.786861 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:10.861859 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:10.861823 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trccd\" (UniqueName: \"kubernetes.io/projected/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b-kube-api-access-trccd\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:11.015216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.015173 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp"] Oct 29 23:52:11.015216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.015221 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-825472-78b9446bcd-b2vdp"] Oct 29 23:52:11.068970 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.068945 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d"] Oct 29 23:52:11.075827 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:52:11.075796 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fb7bea_c53a_4cef_9400_c87c06b54554.slice/crio-18fb8fa5eed9e7dce27c2a6fe266d1b6940b03e3c0e32d74c4277d78a7eb334b WatchSource:0}: Error finding container 18fb8fa5eed9e7dce27c2a6fe266d1b6940b03e3c0e32d74c4277d78a7eb334b: Status 404 returned error can't find the container with id 18fb8fa5eed9e7dce27c2a6fe266d1b6940b03e3c0e32d74c4277d78a7eb334b Oct 29 23:52:11.105204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.105137 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr"] Oct 29 23:52:11.114762 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:52:11.114737 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2959a1_c03a_42aa_999e_135769ce1efc.slice/crio-711ff4aca789c787f037589e6c2f1a24c30f64b2c39d6ee68c10f24929b777cb WatchSource:0}: Error finding container 711ff4aca789c787f037589e6c2f1a24c30f64b2c39d6ee68c10f24929b777cb: Status 404 returned error can't find the container with id 711ff4aca789c787f037589e6c2f1a24c30f64b2c39d6ee68c10f24929b777cb Oct 29 23:52:11.628961 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.628925 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" event={"ID":"e8fb7bea-c53a-4cef-9400-c87c06b54554","Type":"ContainerStarted","Data":"ff01e769139081d773a1d6e3418b1806b76163c7c81abda0c05d2785c634e574"} Oct 29 23:52:11.629388 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.628985 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" event={"ID":"e8fb7bea-c53a-4cef-9400-c87c06b54554","Type":"ContainerStarted","Data":"18fb8fa5eed9e7dce27c2a6fe266d1b6940b03e3c0e32d74c4277d78a7eb334b"} Oct 29 23:52:11.630622 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.630558 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" event={"ID":"2c2959a1-c03a-42aa-999e-135769ce1efc","Type":"ContainerStarted","Data":"9fd813750d260bb1c96d779cbc79ef83e604337ca333e8ac8105db5f9b4b82e6"} Oct 29 23:52:11.630771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.630639 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" event={"ID":"2c2959a1-c03a-42aa-999e-135769ce1efc","Type":"ContainerStarted","Data":"711ff4aca789c787f037589e6c2f1a24c30f64b2c39d6ee68c10f24929b777cb"} Oct 29 23:52:11.673757 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.673701 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" podStartSLOduration=1.673681411 podStartE2EDuration="1.673681411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:52:11.670098762 +0000 UTC m=+4128.786018379" watchObservedRunningTime="2026-10-29 23:52:11.673681411 +0000 UTC m=+4128.789601019" Oct 29 23:52:11.711883 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.711792 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" podStartSLOduration=1.711774772 podStartE2EDuration="1.711774772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:52:11.709881547 +0000 UTC m=+4128.825801153" watchObservedRunningTime="2026-10-29 23:52:11.711774772 +0000 UTC m=+4128.827694379" Oct 29 23:52:11.812535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.812505 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 23:52:11.976040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.975965 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-kube-api-access-pbkbr\" (UniqueName: \"kubernetes.io/projected/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-kube-api-access-pbkbr\") pod \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\" (UID: \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\") " Oct 29 23:52:11.976199 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.976051 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-plugins-config-volume\") pod \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\" (UID: \"1632fbd9-04f1-4a5b-a40f-e778a40a3e55\") " Oct 29 23:52:11.976522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.976491 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-plugins-config-volume" pod "1632fbd9-04f1-4a5b-a40f-e778a40a3e55" (UID: "1632fbd9-04f1-4a5b-a40f-e778a40a3e55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:11.976682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:11.976659 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:12.031126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.031014 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-kube-api-access-pbkbr" pod "1632fbd9-04f1-4a5b-a40f-e778a40a3e55" (UID: "1632fbd9-04f1-4a5b-a40f-e778a40a3e55"). InnerVolumeSpecName "kube-api-access-pbkbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:12.078185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.078156 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbkbr\" (UniqueName: \"kubernetes.io/projected/1632fbd9-04f1-4a5b-a40f-e778a40a3e55-kube-api-access-pbkbr\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:12.647092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.647000 5736 generic.go:356] "Generic (PLEG): container finished" podID="1632fbd9-04f1-4a5b-a40f-e778a40a3e55" containerID="13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46" exitCode=0 Oct 29 23:52:12.647092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.647055 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" Oct 29 23:52:12.647617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.647087 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" event={"ID":"1632fbd9-04f1-4a5b-a40f-e778a40a3e55","Type":"ContainerDied","Data":"13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46"} Oct 29 23:52:12.647617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.647162 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l" event={"ID":"1632fbd9-04f1-4a5b-a40f-e778a40a3e55","Type":"ContainerDied","Data":"20baac6d3f10f66cee7bcb68d505324001ca9a2e98c818641a2f1fe34a4b1584"} Oct 29 23:52:12.647617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.647197 5736 scope.go:122] "RemoveContainer" containerID="13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46" Oct 29 23:52:12.652811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.652786 5736 generic.go:356] "Generic (PLEG): container finished" podID="e8fb7bea-c53a-4cef-9400-c87c06b54554" containerID="ff01e769139081d773a1d6e3418b1806b76163c7c81abda0c05d2785c634e574" exitCode=0 Oct 29 23:52:12.652950 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.652870 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" event={"ID":"e8fb7bea-c53a-4cef-9400-c87c06b54554","Type":"ContainerDied","Data":"ff01e769139081d773a1d6e3418b1806b76163c7c81abda0c05d2785c634e574"} Oct 29 23:52:12.654662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.654636 5736 generic.go:356] "Generic (PLEG): container finished" podID="2c2959a1-c03a-42aa-999e-135769ce1efc" containerID="9fd813750d260bb1c96d779cbc79ef83e604337ca333e8ac8105db5f9b4b82e6" exitCode=0 Oct 29 23:52:12.654800 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.654713 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" event={"ID":"2c2959a1-c03a-42aa-999e-135769ce1efc","Type":"ContainerDied","Data":"9fd813750d260bb1c96d779cbc79ef83e604337ca333e8ac8105db5f9b4b82e6"} Oct 29 23:52:12.659675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.659656 5736 scope.go:122] "RemoveContainer" containerID="13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46" Oct 29 23:52:12.659931 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:12.659909 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46\": container with ID starting with 13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46 not found: ID does not exist" containerID="13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46" Oct 29 23:52:12.659992 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.659937 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46"} err="failed to get container status \"13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46\": rpc error: code = NotFound desc = could not find container \"13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46\": container with ID starting with 13564451d96f064c27a2b8e97d9b9e248ed913e2597a58ebdf750d3fa535ba46 not found: ID does not exist" Oct 29 23:52:12.779957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.779920 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l"] Oct 29 23:52:12.787169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:12.787137 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1ee571-7fcc5cb9c4-2d56l"] Oct 29 23:52:13.004617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:13.004565 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1632fbd9-04f1-4a5b-a40f-e778a40a3e55" path="/var/lib/kubelet/pods/1632fbd9-04f1-4a5b-a40f-e778a40a3e55/volumes" Oct 29 23:52:13.005017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:13.005003 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fed7e2-90a5-41e2-a1e0-22c694c3ef8b" path="/var/lib/kubelet/pods/82fed7e2-90a5-41e2-a1e0-22c694c3ef8b/volumes" Oct 29 23:52:13.813142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:13.813116 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:13.845981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:13.845953 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:14.002382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.002350 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-xks-ca\") pod \"e8fb7bea-c53a-4cef-9400-c87c06b54554\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " Oct 29 23:52:14.002598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.002447 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fb7bea-c53a-4cef-9400-c87c06b54554-kube-api-access-k8lnk\" (UniqueName: \"kubernetes.io/projected/e8fb7bea-c53a-4cef-9400-c87c06b54554-kube-api-access-k8lnk\") pod \"e8fb7bea-c53a-4cef-9400-c87c06b54554\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " Oct 29 23:52:14.002598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.002477 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-xks-ca\") pod \"2c2959a1-c03a-42aa-999e-135769ce1efc\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " Oct 29 23:52:14.002598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.002547 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-openshift-ca\") pod \"e8fb7bea-c53a-4cef-9400-c87c06b54554\" (UID: \"e8fb7bea-c53a-4cef-9400-c87c06b54554\") " Oct 29 23:52:14.002817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.002621 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2959a1-c03a-42aa-999e-135769ce1efc-kube-api-access-cpcmq\" (UniqueName: \"kubernetes.io/projected/2c2959a1-c03a-42aa-999e-135769ce1efc-kube-api-access-cpcmq\") pod \"2c2959a1-c03a-42aa-999e-135769ce1efc\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " Oct 29 23:52:14.002817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.002678 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-openshift-ca\") pod \"2c2959a1-c03a-42aa-999e-135769ce1efc\" (UID: \"2c2959a1-c03a-42aa-999e-135769ce1efc\") " Oct 29 23:52:14.003045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.003014 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-openshift-ca" pod "e8fb7bea-c53a-4cef-9400-c87c06b54554" (UID: "e8fb7bea-c53a-4cef-9400-c87c06b54554"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:14.003045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.003032 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-openshift-ca" pod "2c2959a1-c03a-42aa-999e-135769ce1efc" (UID: "2c2959a1-c03a-42aa-999e-135769ce1efc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:14.003194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.003179 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:14.003261 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.003194 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:14.004753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.004726 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-xks-ca" pod "2c2959a1-c03a-42aa-999e-135769ce1efc" (UID: "2c2959a1-c03a-42aa-999e-135769ce1efc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:14.004878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.004759 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fb7bea-c53a-4cef-9400-c87c06b54554-kube-api-access-k8lnk" pod "e8fb7bea-c53a-4cef-9400-c87c06b54554" (UID: "e8fb7bea-c53a-4cef-9400-c87c06b54554"). InnerVolumeSpecName "kube-api-access-k8lnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:14.004878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.004778 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-xks-ca" pod "e8fb7bea-c53a-4cef-9400-c87c06b54554" (UID: "e8fb7bea-c53a-4cef-9400-c87c06b54554"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:14.005079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.005063 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2959a1-c03a-42aa-999e-135769ce1efc-kube-api-access-cpcmq" pod "2c2959a1-c03a-42aa-999e-135769ce1efc" (UID: "2c2959a1-c03a-42aa-999e-135769ce1efc"). InnerVolumeSpecName "kube-api-access-cpcmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:14.029532 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.029501 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc"] Oct 29 23:52:14.032474 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.032451 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13952795-xpmsc"] Oct 29 23:52:14.034804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.034781 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx"] Oct 29 23:52:14.039997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.039974 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf15c06-ftntx"] Oct 29 23:52:14.104314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.104281 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb7bea-c53a-4cef-9400-c87c06b54554-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:14.104314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.104315 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8lnk\" (UniqueName: \"kubernetes.io/projected/e8fb7bea-c53a-4cef-9400-c87c06b54554-kube-api-access-k8lnk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:14.104510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.104330 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2959a1-c03a-42aa-999e-135769ce1efc-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:14.104510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.104345 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpcmq\" (UniqueName: \"kubernetes.io/projected/2c2959a1-c03a-42aa-999e-135769ce1efc-kube-api-access-cpcmq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:14.673135 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.673102 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" Oct 29 23:52:14.673352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.673104 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d" event={"ID":"e8fb7bea-c53a-4cef-9400-c87c06b54554","Type":"ContainerDied","Data":"18fb8fa5eed9e7dce27c2a6fe266d1b6940b03e3c0e32d74c4277d78a7eb334b"} Oct 29 23:52:14.673352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.673253 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18fb8fa5eed9e7dce27c2a6fe266d1b6940b03e3c0e32d74c4277d78a7eb334b" Oct 29 23:52:14.674702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.674674 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" Oct 29 23:52:14.674813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.674702 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr" event={"ID":"2c2959a1-c03a-42aa-999e-135769ce1efc","Type":"ContainerDied","Data":"711ff4aca789c787f037589e6c2f1a24c30f64b2c39d6ee68c10f24929b777cb"} Oct 29 23:52:14.674813 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:14.674741 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="711ff4aca789c787f037589e6c2f1a24c30f64b2c39d6ee68c10f24929b777cb" Oct 29 23:52:15.014058 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.014018 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f25381-b24f-4bdd-ab20-33f64c0d9ea2" path="/var/lib/kubelet/pods/64f25381-b24f-4bdd-ab20-33f64c0d9ea2/volumes" Oct 29 23:52:15.014659 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.014633 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ba6468-9ab8-4864-b79f-512acb32e038" path="/var/lib/kubelet/pods/84ba6468-9ab8-4864-b79f-512acb32e038/volumes" Oct 29 23:52:15.620334 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.620292 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw"] Oct 29 23:52:15.621230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621202 5736 cpu_manager.go:414] "containerMap: removing container" podUID="82fed7e2-90a5-41e2-a1e0-22c694c3ef8b" containerName="payload-processing" Oct 29 23:52:15.621230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621225 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fed7e2-90a5-41e2-a1e0-22c694c3ef8b" containerName="payload-processing" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621244 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e8fb7bea-c53a-4cef-9400-c87c06b54554" containerName="revoke-keys" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621251 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fb7bea-c53a-4cef-9400-c87c06b54554" containerName="revoke-keys" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621260 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2c2959a1-c03a-42aa-999e-135769ce1efc" containerName="revoke-keys" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621265 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2959a1-c03a-42aa-999e-135769ce1efc" containerName="revoke-keys" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621285 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1632fbd9-04f1-4a5b-a40f-e778a40a3e55" containerName="payload-processing" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621291 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1632fbd9-04f1-4a5b-a40f-e778a40a3e55" containerName="payload-processing" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621401 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fed7e2-90a5-41e2-a1e0-22c694c3ef8b" containerName="payload-processing" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621409 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fb7bea-c53a-4cef-9400-c87c06b54554" containerName="revoke-keys" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621417 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2959a1-c03a-42aa-999e-135769ce1efc" containerName="revoke-keys" Oct 29 23:52:15.621441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.621425 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1632fbd9-04f1-4a5b-a40f-e778a40a3e55" containerName="payload-processing" Oct 29 23:52:15.626352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.626330 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.645746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.645711 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw"] Oct 29 23:52:15.667644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.667606 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:52:15.667793 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.667768 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:52:15.721845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.721805 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fh96\" (UniqueName: \"kubernetes.io/projected/9c4b0c9d-ba19-4661-8ef4-c996470fd040-kube-api-access-7fh96\") pod \"maas-api-revoke-keys-e2e-extmod-b-206095-q92fw\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.722024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.721937 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-206095-q92fw\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.722024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.722014 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-206095-q92fw\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.823344 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.823308 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-206095-q92fw\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.823491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.823376 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-206095-q92fw\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.823566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.823522 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fh96\" (UniqueName: \"kubernetes.io/projected/9c4b0c9d-ba19-4661-8ef4-c996470fd040-kube-api-access-7fh96\") pod \"maas-api-revoke-keys-e2e-extmod-b-206095-q92fw\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.824124 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.824095 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-206095-q92fw\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.825744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.825724 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-206095-q92fw\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.846733 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.846712 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fh96\" (UniqueName: \"kubernetes.io/projected/9c4b0c9d-ba19-4661-8ef4-c996470fd040-kube-api-access-7fh96\") pod \"maas-api-revoke-keys-e2e-extmod-b-206095-q92fw\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:15.940785 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:15.940732 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:16.246983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:16.246912 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw"] Oct 29 23:52:16.249735 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:52:16.249702 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4b0c9d_ba19_4661_8ef4_c996470fd040.slice/crio-06ce94b341a4430b132d360bff322f00d4f711d8f1c51e4122b72ef07c259fec WatchSource:0}: Error finding container 06ce94b341a4430b132d360bff322f00d4f711d8f1c51e4122b72ef07c259fec: Status 404 returned error can't find the container with id 06ce94b341a4430b132d360bff322f00d4f711d8f1c51e4122b72ef07c259fec Oct 29 23:52:16.692500 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:16.692454 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" event={"ID":"9c4b0c9d-ba19-4661-8ef4-c996470fd040","Type":"ContainerStarted","Data":"dda6e4c68e3b72407d46ba353f95190e89a280e1dc8690b159cf58eaa5da8fc5"} Oct 29 23:52:16.692702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:16.692530 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" event={"ID":"9c4b0c9d-ba19-4661-8ef4-c996470fd040","Type":"ContainerStarted","Data":"06ce94b341a4430b132d360bff322f00d4f711d8f1c51e4122b72ef07c259fec"} Oct 29 23:52:16.734085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:16.734034 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" podStartSLOduration=1.7340159210000001 podStartE2EDuration="1.734015921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:52:16.732118071 +0000 UTC m=+4133.848037677" watchObservedRunningTime="2026-10-29 23:52:16.734015921 +0000 UTC m=+4133.849935529" Oct 29 23:52:17.701824 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:17.701741 5736 generic.go:356] "Generic (PLEG): container finished" podID="9c4b0c9d-ba19-4661-8ef4-c996470fd040" containerID="dda6e4c68e3b72407d46ba353f95190e89a280e1dc8690b159cf58eaa5da8fc5" exitCode=0 Oct 29 23:52:17.702204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:17.701833 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" event={"ID":"9c4b0c9d-ba19-4661-8ef4-c996470fd040","Type":"ContainerDied","Data":"dda6e4c68e3b72407d46ba353f95190e89a280e1dc8690b159cf58eaa5da8fc5"} Oct 29 23:52:18.982803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:18.982778 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:19.199377 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.199337 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-xks-ca\") pod \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " Oct 29 23:52:19.199556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.199399 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-openshift-ca\") pod \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " Oct 29 23:52:19.199556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.199447 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4b0c9d-ba19-4661-8ef4-c996470fd040-kube-api-access-7fh96\" (UniqueName: \"kubernetes.io/projected/9c4b0c9d-ba19-4661-8ef4-c996470fd040-kube-api-access-7fh96\") pod \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\" (UID: \"9c4b0c9d-ba19-4661-8ef4-c996470fd040\") " Oct 29 23:52:19.199804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.199772 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-openshift-ca" pod "9c4b0c9d-ba19-4661-8ef4-c996470fd040" (UID: "9c4b0c9d-ba19-4661-8ef4-c996470fd040"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:19.200171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.200150 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:19.201559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.201533 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-xks-ca" pod "9c4b0c9d-ba19-4661-8ef4-c996470fd040" (UID: "9c4b0c9d-ba19-4661-8ef4-c996470fd040"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:19.201735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.201712 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4b0c9d-ba19-4661-8ef4-c996470fd040-kube-api-access-7fh96" pod "9c4b0c9d-ba19-4661-8ef4-c996470fd040" (UID: "9c4b0c9d-ba19-4661-8ef4-c996470fd040"). InnerVolumeSpecName "kube-api-access-7fh96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:19.301035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.300950 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4b0c9d-ba19-4661-8ef4-c996470fd040-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:19.301035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.300983 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fh96\" (UniqueName: \"kubernetes.io/projected/9c4b0c9d-ba19-4661-8ef4-c996470fd040-kube-api-access-7fh96\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:19.720332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.720299 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" Oct 29 23:52:19.720515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.720335 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw" event={"ID":"9c4b0c9d-ba19-4661-8ef4-c996470fd040","Type":"ContainerDied","Data":"06ce94b341a4430b132d360bff322f00d4f711d8f1c51e4122b72ef07c259fec"} Oct 29 23:52:19.720515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.720385 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ce94b341a4430b132d360bff322f00d4f711d8f1c51e4122b72ef07c259fec" Oct 29 23:52:19.987357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.987269 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw"] Oct 29 23:52:19.987820 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:19.987526 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" podUID="234594a5-0c76-43e8-b46e-c296fcb6d9ea" containerName="maas-api" containerID="cri-o://68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3" gracePeriod=30 Oct 29 23:52:20.317337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.317307 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:52:20.457618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.456795 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-maas-api-tls\") pod \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " Oct 29 23:52:20.457618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.456856 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234594a5-0c76-43e8-b46e-c296fcb6d9ea-kube-api-access-zv9g7\" (UniqueName: \"kubernetes.io/projected/234594a5-0c76-43e8-b46e-c296fcb6d9ea-kube-api-access-zv9g7\") pod \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " Oct 29 23:52:20.457618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.456933 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/234594a5-0c76-43e8-b46e-c296fcb6d9ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234594a5-0c76-43e8-b46e-c296fcb6d9ea-openshift-service-ca\") pod \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " Oct 29 23:52:20.457618 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.457027 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-metrics-certs\") pod \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\" (UID: \"234594a5-0c76-43e8-b46e-c296fcb6d9ea\") " Oct 29 23:52:20.482606 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.480934 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234594a5-0c76-43e8-b46e-c296fcb6d9ea-openshift-service-ca" pod "234594a5-0c76-43e8-b46e-c296fcb6d9ea" (UID: "234594a5-0c76-43e8-b46e-c296fcb6d9ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:20.484804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.484661 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234594a5-0c76-43e8-b46e-c296fcb6d9ea-kube-api-access-zv9g7" pod "234594a5-0c76-43e8-b46e-c296fcb6d9ea" (UID: "234594a5-0c76-43e8-b46e-c296fcb6d9ea"). InnerVolumeSpecName "kube-api-access-zv9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:20.484804 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.484753 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-metrics-certs" pod "234594a5-0c76-43e8-b46e-c296fcb6d9ea" (UID: "234594a5-0c76-43e8-b46e-c296fcb6d9ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:20.487210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.487093 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-maas-api-tls" pod "234594a5-0c76-43e8-b46e-c296fcb6d9ea" (UID: "234594a5-0c76-43e8-b46e-c296fcb6d9ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:20.578256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.578182 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.578256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.578210 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234594a5-0c76-43e8-b46e-c296fcb6d9ea-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.578256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.578220 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv9g7\" (UniqueName: \"kubernetes.io/projected/234594a5-0c76-43e8-b46e-c296fcb6d9ea-kube-api-access-zv9g7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.578256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.578229 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234594a5-0c76-43e8-b46e-c296fcb6d9ea-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:20.729614 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.729542 5736 generic.go:356] "Generic (PLEG): container finished" podID="234594a5-0c76-43e8-b46e-c296fcb6d9ea" containerID="68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3" exitCode=0 Oct 29 23:52:20.729832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.729619 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" Oct 29 23:52:20.729832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.729630 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" event={"ID":"234594a5-0c76-43e8-b46e-c296fcb6d9ea","Type":"ContainerDied","Data":"68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3"} Oct 29 23:52:20.729832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.729700 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw" event={"ID":"234594a5-0c76-43e8-b46e-c296fcb6d9ea","Type":"ContainerDied","Data":"23194217f946a0b923ae3484a1cd9fe3a0709a3bc2c22c5e00c63e825670851e"} Oct 29 23:52:20.729832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.729746 5736 scope.go:122] "RemoveContainer" containerID="68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3" Oct 29 23:52:20.741472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.741454 5736 scope.go:122] "RemoveContainer" containerID="68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3" Oct 29 23:52:20.741778 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:20.741754 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3\": container with ID starting with 68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3 not found: ID does not exist" containerID="68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3" Oct 29 23:52:20.741850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.741785 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3"} err="failed to get container status \"68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3\": rpc error: code = NotFound desc = could not find container \"68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3\": container with ID starting with 68f4373de094efa3ad294c573ff59defae16a5b6e252e095754c0047a4611ba3 not found: ID does not exist" Oct 29 23:52:20.797674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.783925 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw"] Oct 29 23:52:20.798351 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.798316 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-37498a-656b865fd4-9x9bw"] Oct 29 23:52:20.893801 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.893722 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w"] Oct 29 23:52:20.897354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:20.897318 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-37498a-67z8w"] Oct 29 23:52:21.006324 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:21.006283 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="234594a5-0c76-43e8-b46e-c296fcb6d9ea" path="/var/lib/kubelet/pods/234594a5-0c76-43e8-b46e-c296fcb6d9ea/volumes" Oct 29 23:52:21.006994 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:21.006972 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e27c60-2a27-4b21-9f47-1572042e5b29" path="/var/lib/kubelet/pods/b4e27c60-2a27-4b21-9f47-1572042e5b29/volumes" Oct 29 23:52:26.021519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:26.021479 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb"] Oct 29 23:52:26.827112 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:26.827067 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:52:26.919128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:26.919091 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg"] Oct 29 23:52:26.930908 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:26.930873 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-825472-2hcbg"] Oct 29 23:52:27.003879 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:27.003850 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c45f48-4dad-4c2c-aba1-e8f799a81dc7" path="/var/lib/kubelet/pods/08c45f48-4dad-4c2c-aba1-e8f799a81dc7/volumes" Oct 29 23:52:28.551696 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:28.551654 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj"] Oct 29 23:52:28.552521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:28.551946 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" podUID="ce05cf00-34b7-4564-b736-950698149d88" containerName="istio-proxy" containerID="cri-o://70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb" gracePeriod=30 Oct 29 23:52:29.201048 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.201005 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57"] Oct 29 23:52:29.201649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.201626 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9c4b0c9d-ba19-4661-8ef4-c996470fd040" containerName="revoke-keys" Oct 29 23:52:29.201649 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.201648 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4b0c9d-ba19-4661-8ef4-c996470fd040" containerName="revoke-keys" Oct 29 23:52:29.201832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.201680 5736 cpu_manager.go:414] "containerMap: removing container" podUID="234594a5-0c76-43e8-b46e-c296fcb6d9ea" containerName="maas-api" Oct 29 23:52:29.201832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.201688 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234594a5-0c76-43e8-b46e-c296fcb6d9ea" containerName="maas-api" Oct 29 23:52:29.201941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.201843 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4b0c9d-ba19-4661-8ef4-c996470fd040" containerName="revoke-keys" Oct 29 23:52:29.201941 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.201863 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="234594a5-0c76-43e8-b46e-c296fcb6d9ea" containerName="maas-api" Oct 29 23:52:29.206758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.206734 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.234920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.234886 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:52:29.234920 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.234903 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:52:29.238022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.237993 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57"] Oct 29 23:52:29.391420 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.391386 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.391630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.391573 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.391789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.391767 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrqz\" (UniqueName: \"kubernetes.io/projected/3872bd34-70f4-404a-8e61-4cf9e2e750de-kube-api-access-rqrqz\") pod \"maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.493264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.493227 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.493524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.493332 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.493524 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.493405 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrqz\" (UniqueName: \"kubernetes.io/projected/3872bd34-70f4-404a-8e61-4cf9e2e750de-kube-api-access-rqrqz\") pod \"maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.494029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.494003 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.495725 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.495696 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.516673 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.516645 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrqz\" (UniqueName: \"kubernetes.io/projected/3872bd34-70f4-404a-8e61-4cf9e2e750de-kube-api-access-rqrqz\") pod \"maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.520435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.520413 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:29.827550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:29.827522 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57"] Oct 29 23:52:29.830283 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:52:29.830248 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3872bd34_70f4_404a_8e61_4cf9e2e750de.slice/crio-a2ed85e3aa39eb8b28658662419616ec0680485afce5e5d96604f36ab5d2c940 WatchSource:0}: Error finding container a2ed85e3aa39eb8b28658662419616ec0680485afce5e5d96604f36ab5d2c940: Status 404 returned error can't find the container with id a2ed85e3aa39eb8b28658662419616ec0680485afce5e5d96604f36ab5d2c940 Oct 29 23:52:30.815651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:30.815617 5736 generic.go:356] "Generic (PLEG): container finished" podID="3872bd34-70f4-404a-8e61-4cf9e2e750de" containerID="e6043f90da2b18a4c57e2f09e77557536ac55a5d5e3a92c11ad1dd478d22028b" exitCode=0 Oct 29 23:52:30.815822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:30.815738 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" event={"ID":"3872bd34-70f4-404a-8e61-4cf9e2e750de","Type":"ContainerDied","Data":"e6043f90da2b18a4c57e2f09e77557536ac55a5d5e3a92c11ad1dd478d22028b"} Oct 29 23:52:30.815822 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:30.815791 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" event={"ID":"3872bd34-70f4-404a-8e61-4cf9e2e750de","Type":"ContainerStarted","Data":"a2ed85e3aa39eb8b28658662419616ec0680485afce5e5d96604f36ab5d2c940"} Oct 29 23:52:31.022719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:31.022614 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" podUID="36aa37e0-2bd6-411e-9483-e79875d4efba" containerName="payload-processing" containerID="cri-o://d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690" gracePeriod=25 Oct 29 23:52:31.503631 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:31.503597 5736 scope.go:122] "RemoveContainer" containerID="d3c01922355044520cf1e8cbd15cf4d84e2bc486618af8dff014e6c8e3b9a4e6" Oct 29 23:52:31.514535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:31.514510 5736 scope.go:122] "RemoveContainer" containerID="377f132a84020570770aa49474c9f1e7087602366ca43a49f3d5cd0e0dce0f8e" Oct 29 23:52:31.551918 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:31.551884 5736 scope.go:122] "RemoveContainer" containerID="578d18be7c1cfa580cb2f58dd85874ef4b1defc25b44b4e9033c52536e472613" Oct 29 23:52:31.970079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:31.970050 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:32.050768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.050727 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-openshift-ca\") pod \"3872bd34-70f4-404a-8e61-4cf9e2e750de\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " Oct 29 23:52:32.051142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.050795 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-xks-ca\") pod \"3872bd34-70f4-404a-8e61-4cf9e2e750de\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " Oct 29 23:52:32.051142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.050855 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3872bd34-70f4-404a-8e61-4cf9e2e750de-kube-api-access-rqrqz\" (UniqueName: \"kubernetes.io/projected/3872bd34-70f4-404a-8e61-4cf9e2e750de-kube-api-access-rqrqz\") pod \"3872bd34-70f4-404a-8e61-4cf9e2e750de\" (UID: \"3872bd34-70f4-404a-8e61-4cf9e2e750de\") " Oct 29 23:52:32.051266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.051169 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-openshift-ca" pod "3872bd34-70f4-404a-8e61-4cf9e2e750de" (UID: "3872bd34-70f4-404a-8e61-4cf9e2e750de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:32.051599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.051554 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:32.052975 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.052952 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-xks-ca" pod "3872bd34-70f4-404a-8e61-4cf9e2e750de" (UID: "3872bd34-70f4-404a-8e61-4cf9e2e750de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:32.053558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.053541 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3872bd34-70f4-404a-8e61-4cf9e2e750de-kube-api-access-rqrqz" pod "3872bd34-70f4-404a-8e61-4cf9e2e750de" (UID: "3872bd34-70f4-404a-8e61-4cf9e2e750de"). InnerVolumeSpecName "kube-api-access-rqrqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:32.153470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.153391 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrqz\" (UniqueName: \"kubernetes.io/projected/3872bd34-70f4-404a-8e61-4cf9e2e750de-kube-api-access-rqrqz\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:32.153470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.153421 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3872bd34-70f4-404a-8e61-4cf9e2e750de-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:32.831237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.831199 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" Oct 29 23:52:32.831425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.831232 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57" event={"ID":"3872bd34-70f4-404a-8e61-4cf9e2e750de","Type":"ContainerDied","Data":"a2ed85e3aa39eb8b28658662419616ec0680485afce5e5d96604f36ab5d2c940"} Oct 29 23:52:32.831425 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:32.831283 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2ed85e3aa39eb8b28658662419616ec0680485afce5e5d96604f36ab5d2c940" Oct 29 23:52:33.812495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.812430 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:52:33.840289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.840218 5736 generic.go:356] "Generic (PLEG): container finished" podID="ce05cf00-34b7-4564-b736-950698149d88" containerID="70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb" exitCode=0 Oct 29 23:52:33.840289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.840262 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" Oct 29 23:52:33.840466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.840266 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" event={"ID":"ce05cf00-34b7-4564-b736-950698149d88","Type":"ContainerDied","Data":"70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb"} Oct 29 23:52:33.840466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.840406 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj" event={"ID":"ce05cf00-34b7-4564-b736-950698149d88","Type":"ContainerDied","Data":"99fe55f1c76ead6461463a21f725825f9a0e06c3d3de769f25a488c820906e72"} Oct 29 23:52:33.840466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.840446 5736 scope.go:122] "RemoveContainer" containerID="70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb" Oct 29 23:52:33.853612 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.853587 5736 scope.go:122] "RemoveContainer" containerID="70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb" Oct 29 23:52:33.853984 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:33.853954 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb\": container with ID starting with 70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb not found: ID does not exist" containerID="70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb" Oct 29 23:52:33.854087 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.853984 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb"} err="failed to get container status \"70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb\": rpc error: code = NotFound desc = could not find container \"70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb\": container with ID starting with 70f3547b8795c890bcccb1712fae5e213af4bb13c06ef94b24c535345a1ddccb not found: ID does not exist" Oct 29 23:52:33.873352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873319 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce05cf00-34b7-4564-b736-950698149d88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05cf00-34b7-4564-b736-950698149d88-istiod-ca-cert\") pod \"ce05cf00-34b7-4564-b736-950698149d88\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " Oct 29 23:52:33.873510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873386 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-certs\") pod \"ce05cf00-34b7-4564-b736-950698149d88\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " Oct 29 23:52:33.873510 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873443 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce05cf00-34b7-4564-b736-950698149d88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05cf00-34b7-4564-b736-950698149d88-istio-podinfo\") pod \"ce05cf00-34b7-4564-b736-950698149d88\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " Oct 29 23:52:33.873702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873516 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-data\") pod \"ce05cf00-34b7-4564-b736-950698149d88\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " Oct 29 23:52:33.873702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873693 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-credential-socket\") pod \"ce05cf00-34b7-4564-b736-950698149d88\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " Oct 29 23:52:33.873836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873737 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-certs" pod "ce05cf00-34b7-4564-b736-950698149d88" (UID: "ce05cf00-34b7-4564-b736-950698149d88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:33.873836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873755 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce05cf00-34b7-4564-b736-950698149d88-istiod-ca-cert" pod "ce05cf00-34b7-4564-b736-950698149d88" (UID: "ce05cf00-34b7-4564-b736-950698149d88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:33.873836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873748 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-envoy\") pod \"ce05cf00-34b7-4564-b736-950698149d88\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " Oct 29 23:52:33.873997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873845 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-istio-token\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-istio-token\") pod \"ce05cf00-34b7-4564-b736-950698149d88\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " Oct 29 23:52:33.873997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873891 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-socket\") pod \"ce05cf00-34b7-4564-b736-950698149d88\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " Oct 29 23:52:33.873997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.873975 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-credential-socket" pod "ce05cf00-34b7-4564-b736-950698149d88" (UID: "ce05cf00-34b7-4564-b736-950698149d88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:33.874182 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.874031 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-kube-api-access-d7dkv\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-kube-api-access-d7dkv\") pod \"ce05cf00-34b7-4564-b736-950698149d88\" (UID: \"ce05cf00-34b7-4564-b736-950698149d88\") " Oct 29 23:52:33.874240 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.874173 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-socket" pod "ce05cf00-34b7-4564-b736-950698149d88" (UID: "ce05cf00-34b7-4564-b736-950698149d88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:33.874745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.874550 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:33.874745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.874599 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:33.874745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.874616 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05cf00-34b7-4564-b736-950698149d88-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:33.874745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.874630 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:33.874745 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.874550 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-data" pod "ce05cf00-34b7-4564-b736-950698149d88" (UID: "ce05cf00-34b7-4564-b736-950698149d88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:33.876767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.876719 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-envoy" pod "ce05cf00-34b7-4564-b736-950698149d88" (UID: "ce05cf00-34b7-4564-b736-950698149d88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:33.876892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.876767 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-kube-api-access-d7dkv" pod "ce05cf00-34b7-4564-b736-950698149d88" (UID: "ce05cf00-34b7-4564-b736-950698149d88"). InnerVolumeSpecName "kube-api-access-d7dkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:33.877458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.876746 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-istio-token" pod "ce05cf00-34b7-4564-b736-950698149d88" (UID: "ce05cf00-34b7-4564-b736-950698149d88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:33.878130 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.876939 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce05cf00-34b7-4564-b736-950698149d88-istio-podinfo" pod "ce05cf00-34b7-4564-b736-950698149d88" (UID: "ce05cf00-34b7-4564-b736-950698149d88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:52:33.976031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.975976 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:33.976031 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.976025 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:33.976249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.976067 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7dkv\" (UniqueName: \"kubernetes.io/projected/ce05cf00-34b7-4564-b736-950698149d88-kube-api-access-d7dkv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:33.976249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.976081 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05cf00-34b7-4564-b736-950698149d88-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:33.976249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:33.976093 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05cf00-34b7-4564-b736-950698149d88-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:34.188747 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:34.188706 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj"] Oct 29 23:52:34.200684 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:34.200650 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-37498a-openshift-default-b4b666d47-2lqqj"] Oct 29 23:52:35.003661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:35.003576 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce05cf00-34b7-4564-b736-950698149d88" path="/var/lib/kubelet/pods/ce05cf00-34b7-4564-b736-950698149d88/volumes" Oct 29 23:52:35.328894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:35.328795 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" podUID="36aa37e0-2bd6-411e-9483-e79875d4efba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:42.817766 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:42.808704 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn"] Oct 29 23:52:42.825981 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:42.825941 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1ee571-d2fjn"] Oct 29 23:52:43.003424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:43.003386 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2dbac38-12ca-4340-98b9-e677db183f0e" path="/var/lib/kubelet/pods/a2dbac38-12ca-4340-98b9-e677db183f0e/volumes" Oct 29 23:52:43.625109 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:43.625064 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x"] Oct 29 23:52:43.625354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:43.625329 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" containerID="cri-o://8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0" gracePeriod=30 Oct 29 23:52:43.635617 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:43.635554 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9"] Oct 29 23:52:43.636269 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:43.636237 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" containerID="cri-o://a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd" gracePeriod=30 Oct 29 23:52:43.647571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:43.647537 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w"] Oct 29 23:52:43.647868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:43.647818 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" containerID="cri-o://441bd3e3594a0ce808e6b7acfdf1f3ea99226f2e0be730cfc78aef21ba285491" gracePeriod=30 Oct 29 23:52:44.201634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.201593 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj"] Oct 29 23:52:44.202165 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.202148 5736 cpu_manager.go:414] "containerMap: removing container" podUID="3872bd34-70f4-404a-8e61-4cf9e2e750de" containerName="revoke-keys" Oct 29 23:52:44.202216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.202165 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3872bd34-70f4-404a-8e61-4cf9e2e750de" containerName="revoke-keys" Oct 29 23:52:44.202216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.202187 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ce05cf00-34b7-4564-b736-950698149d88" containerName="istio-proxy" Oct 29 23:52:44.202216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.202193 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce05cf00-34b7-4564-b736-950698149d88" containerName="istio-proxy" Oct 29 23:52:44.202322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.202277 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="3872bd34-70f4-404a-8e61-4cf9e2e750de" containerName="revoke-keys" Oct 29 23:52:44.202322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.202290 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce05cf00-34b7-4564-b736-950698149d88" containerName="istio-proxy" Oct 29 23:52:44.207417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.207397 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.224216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.224194 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:52:44.224354 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.224256 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:52:44.227519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.227493 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj"] Oct 29 23:52:44.341393 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.341345 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.341611 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.341488 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.341682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.341640 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742gp\" (UniqueName: \"kubernetes.io/projected/f13e3e46-61b1-436e-87c5-1c06250003b1-kube-api-access-742gp\") pod \"maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.442936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.442899 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.443118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.443007 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742gp\" (UniqueName: \"kubernetes.io/projected/f13e3e46-61b1-436e-87c5-1c06250003b1-kube-api-access-742gp\") pod \"maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.443118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.443074 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.443845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.443819 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.445559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.445532 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.466216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.466184 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-742gp\" (UniqueName: \"kubernetes.io/projected/f13e3e46-61b1-436e-87c5-1c06250003b1-kube-api-access-742gp\") pod \"maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.521089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.521057 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:44.842778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.842741 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj"] Oct 29 23:52:44.935059 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:44.935026 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" event={"ID":"f13e3e46-61b1-436e-87c5-1c06250003b1","Type":"ContainerStarted","Data":"041d75cfae92a3337115558754907f1e569f040217961fc543a6082e52bac918"} Oct 29 23:52:45.329153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:45.329110 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" podUID="36aa37e0-2bd6-411e-9483-e79875d4efba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:52:45.944276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:45.944192 5736 generic.go:356] "Generic (PLEG): container finished" podID="f13e3e46-61b1-436e-87c5-1c06250003b1" containerID="5f8c609bc3ed1a2445f8bab7a9dea8d2c738a5d5c373b36eea816f57eeaec079" exitCode=0 Oct 29 23:52:45.944276 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:45.944235 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" event={"ID":"f13e3e46-61b1-436e-87c5-1c06250003b1","Type":"ContainerDied","Data":"5f8c609bc3ed1a2445f8bab7a9dea8d2c738a5d5c373b36eea816f57eeaec079"} Oct 29 23:52:47.107564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.107536 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:47.180755 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.180635 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57"] Oct 29 23:52:47.183299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.183268 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-adc7d7-djv57"] Oct 29 23:52:47.283104 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.283073 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-openshift-ca\") pod \"f13e3e46-61b1-436e-87c5-1c06250003b1\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " Oct 29 23:52:47.283278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.283193 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-xks-ca\") pod \"f13e3e46-61b1-436e-87c5-1c06250003b1\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " Oct 29 23:52:47.283278 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.283238 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f13e3e46-61b1-436e-87c5-1c06250003b1-kube-api-access-742gp\" (UniqueName: \"kubernetes.io/projected/f13e3e46-61b1-436e-87c5-1c06250003b1-kube-api-access-742gp\") pod \"f13e3e46-61b1-436e-87c5-1c06250003b1\" (UID: \"f13e3e46-61b1-436e-87c5-1c06250003b1\") " Oct 29 23:52:47.283508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.283478 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-openshift-ca" pod "f13e3e46-61b1-436e-87c5-1c06250003b1" (UID: "f13e3e46-61b1-436e-87c5-1c06250003b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:47.283689 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.283666 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:47.285248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.285224 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-xks-ca" pod "f13e3e46-61b1-436e-87c5-1c06250003b1" (UID: "f13e3e46-61b1-436e-87c5-1c06250003b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:52:47.285332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.285283 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13e3e46-61b1-436e-87c5-1c06250003b1-kube-api-access-742gp" pod "f13e3e46-61b1-436e-87c5-1c06250003b1" (UID: "f13e3e46-61b1-436e-87c5-1c06250003b1"). InnerVolumeSpecName "kube-api-access-742gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:47.384519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.384485 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-742gp\" (UniqueName: \"kubernetes.io/projected/f13e3e46-61b1-436e-87c5-1c06250003b1-kube-api-access-742gp\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:47.384519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.384514 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13e3e46-61b1-436e-87c5-1c06250003b1-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:47.959811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.959776 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" Oct 29 23:52:47.959999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.959816 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj" event={"ID":"f13e3e46-61b1-436e-87c5-1c06250003b1","Type":"ContainerDied","Data":"041d75cfae92a3337115558754907f1e569f040217961fc543a6082e52bac918"} Oct 29 23:52:47.959999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:47.959870 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="041d75cfae92a3337115558754907f1e569f040217961fc543a6082e52bac918" Oct 29 23:52:48.953574 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.953549 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:52:48.957047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.957023 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:52:48.968223 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.968195 5736 generic.go:356] "Generic (PLEG): container finished" podID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerID="a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd" exitCode=0 Oct 29 23:52:48.968352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.968233 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" Oct 29 23:52:48.968352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.968276 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" event={"ID":"96ea5a9d-905f-45e0-87c0-1625cb7c2618","Type":"ContainerDied","Data":"a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd"} Oct 29 23:52:48.968352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.968332 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9" event={"ID":"96ea5a9d-905f-45e0-87c0-1625cb7c2618","Type":"ContainerDied","Data":"674ecd409db9756004a9a9ff28f6f69fe569b8fa05642e58c1f8cfef94810fdb"} Oct 29 23:52:48.968486 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.968362 5736 scope.go:122] "RemoveContainer" containerID="a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd" Oct 29 23:52:48.969931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.969904 5736 generic.go:356] "Generic (PLEG): container finished" podID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerID="8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0" exitCode=0 Oct 29 23:52:48.970053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.969966 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" Oct 29 23:52:48.970053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.969990 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" event={"ID":"d2bc7bb9-500f-49c8-a4dd-8011592e7de2","Type":"ContainerDied","Data":"8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0"} Oct 29 23:52:48.970053 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.970033 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x" event={"ID":"d2bc7bb9-500f-49c8-a4dd-8011592e7de2","Type":"ContainerDied","Data":"1aa6fa5716356cb09d3fdc33795bebc9e5f1c1a30bd40459597b9a0505f841a6"} Oct 29 23:52:48.971712 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.971687 5736 generic.go:356] "Generic (PLEG): container finished" podID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerID="441bd3e3594a0ce808e6b7acfdf1f3ea99226f2e0be730cfc78aef21ba285491" exitCode=0 Oct 29 23:52:48.971815 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.971722 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" event={"ID":"ff57e2f5-af27-42fd-a5ed-0bd386e72344","Type":"ContainerDied","Data":"441bd3e3594a0ce808e6b7acfdf1f3ea99226f2e0be730cfc78aef21ba285491"} Oct 29 23:52:48.979889 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.979865 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:52:48.980200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.980182 5736 scope.go:122] "RemoveContainer" containerID="a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd" Oct 29 23:52:48.980459 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:48.980439 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd\": container with ID starting with a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd not found: ID does not exist" containerID="a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd" Oct 29 23:52:48.980539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.980477 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd"} err="failed to get container status \"a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd\": rpc error: code = NotFound desc = could not find container \"a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd\": container with ID starting with a9c1fe8239c549ef49e7f2d4c29c1ae861bd6ed20d03858cc86cc3e04b2ff3bd not found: ID does not exist" Oct 29 23:52:48.980539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.980494 5736 scope.go:122] "RemoveContainer" containerID="8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0" Oct 29 23:52:48.989925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.989908 5736 scope.go:122] "RemoveContainer" containerID="8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0" Oct 29 23:52:48.990186 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:48.990168 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0\": container with ID starting with 8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0 not found: ID does not exist" containerID="8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0" Oct 29 23:52:48.990257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:48.990191 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0"} err="failed to get container status \"8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0\": rpc error: code = NotFound desc = could not find container \"8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0\": container with ID starting with 8dcfc144eebd644a082ae72f6a905d07701db3db93c06291678bb1872c9723a0 not found: ID does not exist" Oct 29 23:52:49.022431 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.018394 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3872bd34-70f4-404a-8e61-4cf9e2e750de" path="/var/lib/kubelet/pods/3872bd34-70f4-404a-8e61-4cf9e2e750de/volumes" Oct 29 23:52:49.103928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.103880 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-credential-socket\") pod \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " Oct 29 23:52:49.104156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.103936 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-data\") pod \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " Oct 29 23:52:49.104156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.103998 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-podinfo\") pod \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " Oct 29 23:52:49.104156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104037 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-kube-api-access-r95c8\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-kube-api-access-r95c8\") pod \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " Oct 29 23:52:49.104156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104066 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-token\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-token\") pod \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " Oct 29 23:52:49.104156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104122 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-certs\") pod \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " Oct 29 23:52:49.104156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104148 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-envoy\") pod \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104175 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-credential-socket" pod "d2bc7bb9-500f-49c8-a4dd-8011592e7de2" (UID: "d2bc7bb9-500f-49c8-a4dd-8011592e7de2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104205 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istiod-ca-cert\") pod \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104233 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-data\") pod \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104302 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-data" pod "96ea5a9d-905f-45e0-87c0-1625cb7c2618" (UID: "96ea5a9d-905f-45e0-87c0-1625cb7c2618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104360 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-token\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-token\") pod \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104392 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-podinfo\") pod \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104416 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-certs" pod "ff57e2f5-af27-42fd-a5ed-0bd386e72344" (UID: "ff57e2f5-af27-42fd-a5ed-0bd386e72344"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104423 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-envoy\") pod \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104460 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istiod-ca-cert\") pod \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " Oct 29 23:52:49.104538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104510 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-credential-socket\") pod \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104548 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istiod-ca-cert\") pod \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104610 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-certs\") pod \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104639 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-kube-api-access-k8xr6\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-kube-api-access-k8xr6\") pod \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104687 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-credential-socket\") pod \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104717 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-socket\") pod \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104752 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-socket\") pod \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104803 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-token\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-token\") pod \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\" (UID: \"96ea5a9d-905f-45e0-87c0-1625cb7c2618\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104884 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-data\") pod \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104938 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-kube-api-access-xd2l2\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-kube-api-access-xd2l2\") pod \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104966 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-socket\") pod \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104997 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-podinfo\") pod \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.105031 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-certs\") pod \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\" (UID: \"d2bc7bb9-500f-49c8-a4dd-8011592e7de2\") " Oct 29 23:52:49.105222 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.105066 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-envoy\") pod \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\" (UID: \"ff57e2f5-af27-42fd-a5ed-0bd386e72344\") " Oct 29 23:52:49.105995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.105696 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.105995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.105721 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.105995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.105737 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.107018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.104612 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-data" pod "ff57e2f5-af27-42fd-a5ed-0bd386e72344" (UID: "ff57e2f5-af27-42fd-a5ed-0bd386e72344"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.107018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.105068 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istiod-ca-cert" pod "ff57e2f5-af27-42fd-a5ed-0bd386e72344" (UID: "ff57e2f5-af27-42fd-a5ed-0bd386e72344"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:49.107018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.105338 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istiod-ca-cert" pod "d2bc7bb9-500f-49c8-a4dd-8011592e7de2" (UID: "d2bc7bb9-500f-49c8-a4dd-8011592e7de2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:49.107018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.106423 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-socket" pod "ff57e2f5-af27-42fd-a5ed-0bd386e72344" (UID: "ff57e2f5-af27-42fd-a5ed-0bd386e72344"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.107018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.106629 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-credential-socket" pod "96ea5a9d-905f-45e0-87c0-1625cb7c2618" (UID: "96ea5a9d-905f-45e0-87c0-1625cb7c2618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.107374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.107040 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-credential-socket" pod "ff57e2f5-af27-42fd-a5ed-0bd386e72344" (UID: "ff57e2f5-af27-42fd-a5ed-0bd386e72344"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.107374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.107240 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-socket" pod "96ea5a9d-905f-45e0-87c0-1625cb7c2618" (UID: "96ea5a9d-905f-45e0-87c0-1625cb7c2618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.107374 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.107355 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-token" pod "d2bc7bb9-500f-49c8-a4dd-8011592e7de2" (UID: "d2bc7bb9-500f-49c8-a4dd-8011592e7de2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.107548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.107398 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-certs" pod "96ea5a9d-905f-45e0-87c0-1625cb7c2618" (UID: "96ea5a9d-905f-45e0-87c0-1625cb7c2618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.107548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.107451 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-envoy" pod "d2bc7bb9-500f-49c8-a4dd-8011592e7de2" (UID: "d2bc7bb9-500f-49c8-a4dd-8011592e7de2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.108163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.107943 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-socket" pod "d2bc7bb9-500f-49c8-a4dd-8011592e7de2" (UID: "d2bc7bb9-500f-49c8-a4dd-8011592e7de2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.108163 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.107949 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istiod-ca-cert" pod "96ea5a9d-905f-45e0-87c0-1625cb7c2618" (UID: "96ea5a9d-905f-45e0-87c0-1625cb7c2618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:49.108346 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.108222 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-data" pod "d2bc7bb9-500f-49c8-a4dd-8011592e7de2" (UID: "d2bc7bb9-500f-49c8-a4dd-8011592e7de2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.108415 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.108390 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-podinfo" pod "96ea5a9d-905f-45e0-87c0-1625cb7c2618" (UID: "96ea5a9d-905f-45e0-87c0-1625cb7c2618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:52:49.108564 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.108527 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-certs" pod "d2bc7bb9-500f-49c8-a4dd-8011592e7de2" (UID: "d2bc7bb9-500f-49c8-a4dd-8011592e7de2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.109143 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.109112 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-token" pod "96ea5a9d-905f-45e0-87c0-1625cb7c2618" (UID: "96ea5a9d-905f-45e0-87c0-1625cb7c2618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.109555 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.109528 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-envoy" pod "ff57e2f5-af27-42fd-a5ed-0bd386e72344" (UID: "ff57e2f5-af27-42fd-a5ed-0bd386e72344"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.109726 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.109704 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-kube-api-access-r95c8" pod "d2bc7bb9-500f-49c8-a4dd-8011592e7de2" (UID: "d2bc7bb9-500f-49c8-a4dd-8011592e7de2"). InnerVolumeSpecName "kube-api-access-r95c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.110142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.110116 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-envoy" pod "96ea5a9d-905f-45e0-87c0-1625cb7c2618" (UID: "96ea5a9d-905f-45e0-87c0-1625cb7c2618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:49.110249 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.110177 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-podinfo" pod "ff57e2f5-af27-42fd-a5ed-0bd386e72344" (UID: "ff57e2f5-af27-42fd-a5ed-0bd386e72344"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:52:49.110370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.110348 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-token" pod "ff57e2f5-af27-42fd-a5ed-0bd386e72344" (UID: "ff57e2f5-af27-42fd-a5ed-0bd386e72344"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.111016 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.110993 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-kube-api-access-k8xr6" pod "96ea5a9d-905f-45e0-87c0-1625cb7c2618" (UID: "96ea5a9d-905f-45e0-87c0-1625cb7c2618"). InnerVolumeSpecName "kube-api-access-k8xr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.111317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.111300 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-kube-api-access-xd2l2" pod "ff57e2f5-af27-42fd-a5ed-0bd386e72344" (UID: "ff57e2f5-af27-42fd-a5ed-0bd386e72344"). InnerVolumeSpecName "kube-api-access-xd2l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:49.111925 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.111908 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-podinfo" pod "d2bc7bb9-500f-49c8-a4dd-8011592e7de2" (UID: "d2bc7bb9-500f-49c8-a4dd-8011592e7de2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:52:49.206922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206824 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.206922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206861 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2l2\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-kube-api-access-xd2l2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.206922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206876 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.206922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206889 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.206922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206902 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.206922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206914 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.206922 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206927 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206938 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r95c8\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-kube-api-access-r95c8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206951 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206963 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206973 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2bc7bb9-500f-49c8-a4dd-8011592e7de2-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206983 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.206994 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207005 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207016 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207029 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff57e2f5-af27-42fd-a5ed-0bd386e72344-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207042 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207054 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207068 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207079 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8xr6\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-kube-api-access-k8xr6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207091 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207103 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff57e2f5-af27-42fd-a5ed-0bd386e72344-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207117 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ea5a9d-905f-45e0-87c0-1625cb7c2618-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.207345 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.207130 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ea5a9d-905f-45e0-87c0-1625cb7c2618-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:49.317523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.317482 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9"] Oct 29 23:52:49.321419 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.321383 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-gx5g9"] Oct 29 23:52:49.356679 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.356645 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x"] Oct 29 23:52:49.362097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.362065 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-nbv2x"] Oct 29 23:52:49.983735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.983706 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" Oct 29 23:52:49.984200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.983701 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w" event={"ID":"ff57e2f5-af27-42fd-a5ed-0bd386e72344","Type":"ContainerDied","Data":"30595d17ebb53b51badef740ed1cc4e3ffb75760c98ac13caedee04236678d0f"} Oct 29 23:52:49.984200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:49.983835 5736 scope.go:122] "RemoveContainer" containerID="441bd3e3594a0ce808e6b7acfdf1f3ea99226f2e0be730cfc78aef21ba285491" Oct 29 23:52:50.036386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:50.036352 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w"] Oct 29 23:52:50.047823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:50.047792 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-wbm8w"] Oct 29 23:52:50.140074 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:50.140037 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd"] Oct 29 23:52:50.140376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:50.140345 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" containerID="cri-o://a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe" gracePeriod=30 Oct 29 23:52:51.003362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.003322 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" path="/var/lib/kubelet/pods/96ea5a9d-905f-45e0-87c0-1625cb7c2618/volumes" Oct 29 23:52:51.003844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.003827 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" path="/var/lib/kubelet/pods/d2bc7bb9-500f-49c8-a4dd-8011592e7de2/volumes" Oct 29 23:52:51.004257 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.004244 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" path="/var/lib/kubelet/pods/ff57e2f5-af27-42fd-a5ed-0bd386e72344/volumes" Oct 29 23:52:51.251032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.250996 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw"] Oct 29 23:52:51.269658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.267805 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-206095-q92fw"] Oct 29 23:52:51.299121 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.299096 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:52:51.330032 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.330003 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36aa37e0-2bd6-411e-9483-e79875d4efba-kube-api-access-hl6p8\" (UniqueName: \"kubernetes.io/projected/36aa37e0-2bd6-411e-9483-e79875d4efba-kube-api-access-hl6p8\") pod \"36aa37e0-2bd6-411e-9483-e79875d4efba\" (UID: \"36aa37e0-2bd6-411e-9483-e79875d4efba\") " Oct 29 23:52:51.330147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.330134 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36aa37e0-2bd6-411e-9483-e79875d4efba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36aa37e0-2bd6-411e-9483-e79875d4efba-plugins-config-volume\") pod \"36aa37e0-2bd6-411e-9483-e79875d4efba\" (UID: \"36aa37e0-2bd6-411e-9483-e79875d4efba\") " Oct 29 23:52:51.330545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.330517 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36aa37e0-2bd6-411e-9483-e79875d4efba-plugins-config-volume" pod "36aa37e0-2bd6-411e-9483-e79875d4efba" (UID: "36aa37e0-2bd6-411e-9483-e79875d4efba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:51.332256 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.332225 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36aa37e0-2bd6-411e-9483-e79875d4efba-kube-api-access-hl6p8" pod "36aa37e0-2bd6-411e-9483-e79875d4efba" (UID: "36aa37e0-2bd6-411e-9483-e79875d4efba"). InnerVolumeSpecName "kube-api-access-hl6p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:51.431304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.431266 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl6p8\" (UniqueName: \"kubernetes.io/projected/36aa37e0-2bd6-411e-9483-e79875d4efba-kube-api-access-hl6p8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:51.431304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:51.431298 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36aa37e0-2bd6-411e-9483-e79875d4efba-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:52.000661 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:52.000624 5736 generic.go:356] "Generic (PLEG): container finished" podID="36aa37e0-2bd6-411e-9483-e79875d4efba" containerID="d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690" exitCode=0 Oct 29 23:52:52.000855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:52.000662 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" Oct 29 23:52:52.000855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:52.000702 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" event={"ID":"36aa37e0-2bd6-411e-9483-e79875d4efba","Type":"ContainerDied","Data":"d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690"} Oct 29 23:52:52.000855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:52.000748 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb" event={"ID":"36aa37e0-2bd6-411e-9483-e79875d4efba","Type":"ContainerDied","Data":"3327ae4ac6e1076e84ca5a9a874fe0b23dc07b3ec2dafff51747da6862b37244"} Oct 29 23:52:52.000855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:52.000772 5736 scope.go:122] "RemoveContainer" containerID="d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690" Oct 29 23:52:52.012018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:52.011998 5736 scope.go:122] "RemoveContainer" containerID="d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690" Oct 29 23:52:52.012305 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:52.012260 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690\": container with ID starting with d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690 not found: ID does not exist" containerID="d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690" Oct 29 23:52:52.012305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:52.012285 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690"} err="failed to get container status \"d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690\": rpc error: code = NotFound desc = could not find container \"d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690\": container with ID starting with d80710382c769d4810f10bf22c5122243add39b6e42ad1067ebdc0649431f690 not found: ID does not exist" Oct 29 23:52:52.052026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:52.051996 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb"] Oct 29 23:52:52.063931 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:52.063903 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-206095-7775f4646-tthgb"] Oct 29 23:52:53.003371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:53.003331 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36aa37e0-2bd6-411e-9483-e79875d4efba" path="/var/lib/kubelet/pods/36aa37e0-2bd6-411e-9483-e79875d4efba/volumes" Oct 29 23:52:53.004097 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:53.004077 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4b0c9d-ba19-4661-8ef4-c996470fd040" path="/var/lib/kubelet/pods/9c4b0c9d-ba19-4661-8ef4-c996470fd040/volumes" Oct 29 23:52:53.565044 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:53.565007 5736 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:52:53.565446 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:53.565073 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:52:55.414850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.414791 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 23:52:55.485703 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.485671 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-kube-api-access-wwp7p\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-kube-api-access-wwp7p\") pod \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " Oct 29 23:52:55.485906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.485836 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-token\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-token\") pod \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " Oct 29 23:52:55.485906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.485889 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istiod-ca-cert\") pod \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " Oct 29 23:52:55.486200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.485921 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-socket\") pod \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " Oct 29 23:52:55.486200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.485986 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-podinfo\") pod \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " Oct 29 23:52:55.486200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.486066 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-credential-socket\") pod \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " Oct 29 23:52:55.486200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.486117 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-data\") pod \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " Oct 29 23:52:55.486200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.486156 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-envoy\") pod \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " Oct 29 23:52:55.486200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.486606 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-certs\") pod \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\" (UID: \"8d260f4a-dfb6-49d7-b59b-5cdef42f6686\") " Oct 29 23:52:55.486200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.486747 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-socket" pod "8d260f4a-dfb6-49d7-b59b-5cdef42f6686" (UID: "8d260f4a-dfb6-49d7-b59b-5cdef42f6686"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:55.487376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.487050 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-data" pod "8d260f4a-dfb6-49d7-b59b-5cdef42f6686" (UID: "8d260f4a-dfb6-49d7-b59b-5cdef42f6686"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:55.487376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.487103 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-credential-socket" pod "8d260f4a-dfb6-49d7-b59b-5cdef42f6686" (UID: "8d260f4a-dfb6-49d7-b59b-5cdef42f6686"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:55.487811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.487744 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istiod-ca-cert" pod "8d260f4a-dfb6-49d7-b59b-5cdef42f6686" (UID: "8d260f4a-dfb6-49d7-b59b-5cdef42f6686"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:52:55.488274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.487957 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-certs" pod "8d260f4a-dfb6-49d7-b59b-5cdef42f6686" (UID: "8d260f4a-dfb6-49d7-b59b-5cdef42f6686"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:55.488513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.488428 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:55.488513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.488454 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:55.488513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.488468 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:55.488513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.488480 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:55.488513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.488493 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:55.489878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.489852 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-envoy" pod "8d260f4a-dfb6-49d7-b59b-5cdef42f6686" (UID: "8d260f4a-dfb6-49d7-b59b-5cdef42f6686"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:52:55.491191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.491162 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-kube-api-access-wwp7p" pod "8d260f4a-dfb6-49d7-b59b-5cdef42f6686" (UID: "8d260f4a-dfb6-49d7-b59b-5cdef42f6686"). InnerVolumeSpecName "kube-api-access-wwp7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:55.491301 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.491222 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-token" pod "8d260f4a-dfb6-49d7-b59b-5cdef42f6686" (UID: "8d260f4a-dfb6-49d7-b59b-5cdef42f6686"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:52:55.494698 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.494669 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-podinfo" pod "8d260f4a-dfb6-49d7-b59b-5cdef42f6686" (UID: "8d260f4a-dfb6-49d7-b59b-5cdef42f6686"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:52:55.590115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.590079 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:55.590115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.590109 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:55.590115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.590121 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwp7p\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-kube-api-access-wwp7p\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:55.590360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:55.590130 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d260f4a-dfb6-49d7-b59b-5cdef42f6686-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:52:56.030860 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:56.030822 5736 generic.go:356] "Generic (PLEG): container finished" podID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerID="a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe" exitCode=0 Oct 29 23:52:56.031047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:56.030874 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" Oct 29 23:52:56.031047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:56.030883 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" event={"ID":"8d260f4a-dfb6-49d7-b59b-5cdef42f6686","Type":"ContainerDied","Data":"a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe"} Oct 29 23:52:56.031047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:56.030927 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd" event={"ID":"8d260f4a-dfb6-49d7-b59b-5cdef42f6686","Type":"ContainerDied","Data":"6e6d797bdffb44406e61fde931f2c0ea66223d510db61dce4306a7269269a59a"} Oct 29 23:52:56.031047 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:56.030947 5736 scope.go:122] "RemoveContainer" containerID="a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe" Oct 29 23:52:56.042593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:56.042562 5736 scope.go:122] "RemoveContainer" containerID="a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe" Oct 29 23:52:56.042877 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:56.042852 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe\": container with ID starting with a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe not found: ID does not exist" containerID="a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe" Oct 29 23:52:56.042936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:56.042889 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe"} err="failed to get container status \"a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe\": rpc error: code = NotFound desc = could not find container \"a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe\": container with ID starting with a63d1e850bf6526dbb9fb840d8a8cfa333e2e8e4d0e668ac89f648a417b527fe not found: ID does not exist" Oct 29 23:52:56.095645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:56.095609 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd"] Oct 29 23:52:56.107197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:56.107167 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-55fwd"] Oct 29 23:52:56.824810 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:52:56.824767 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:52:57.003906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:57.003873 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" path="/var/lib/kubelet/pods/8d260f4a-dfb6-49d7-b59b-5cdef42f6686/volumes" Oct 29 23:52:58.368898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.368840 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk"] Oct 29 23:52:58.369549 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.369492 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" containerID="cri-o://ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb" gracePeriod=30 Oct 29 23:52:58.944838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.944793 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx"] Oct 29 23:52:58.945413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945389 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" Oct 29 23:52:58.945525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945414 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" Oct 29 23:52:58.945525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945429 5736 cpu_manager.go:414] "containerMap: removing container" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" Oct 29 23:52:58.945525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945437 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" Oct 29 23:52:58.945525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945466 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f13e3e46-61b1-436e-87c5-1c06250003b1" containerName="revoke-keys" Oct 29 23:52:58.945525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945474 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13e3e46-61b1-436e-87c5-1c06250003b1" containerName="revoke-keys" Oct 29 23:52:58.945525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945498 5736 cpu_manager.go:414] "containerMap: removing container" podUID="36aa37e0-2bd6-411e-9483-e79875d4efba" containerName="payload-processing" Oct 29 23:52:58.945525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945505 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36aa37e0-2bd6-411e-9483-e79875d4efba" containerName="payload-processing" Oct 29 23:52:58.945525 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945524 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" Oct 29 23:52:58.946015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945532 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" Oct 29 23:52:58.946015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945543 5736 cpu_manager.go:414] "containerMap: removing container" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" Oct 29 23:52:58.946015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945553 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" Oct 29 23:52:58.946015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945687 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13e3e46-61b1-436e-87c5-1c06250003b1" containerName="revoke-keys" Oct 29 23:52:58.946015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945704 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="36aa37e0-2bd6-411e-9483-e79875d4efba" containerName="payload-processing" Oct 29 23:52:58.946015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945717 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff57e2f5-af27-42fd-a5ed-0bd386e72344" containerName="istio-proxy" Oct 29 23:52:58.946015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945727 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2bc7bb9-500f-49c8-a4dd-8011592e7de2" containerName="istio-proxy" Oct 29 23:52:58.946015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945744 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d260f4a-dfb6-49d7-b59b-5cdef42f6686" containerName="istio-proxy" Oct 29 23:52:58.946015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.945753 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ea5a9d-905f-45e0-87c0-1625cb7c2618" containerName="istio-proxy" Oct 29 23:52:58.950489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.950465 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:58.970479 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:58.970450 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx"] Oct 29 23:52:59.018831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.018795 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:52:59.018998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.018869 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:52:59.032046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.032013 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98l7w\" (UniqueName: \"kubernetes.io/projected/9a3de67b-eabc-4b9e-aebf-9904422ba430-kube-api-access-98l7w\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.032162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.032056 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.032217 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.032161 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.133185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.133137 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98l7w\" (UniqueName: \"kubernetes.io/projected/9a3de67b-eabc-4b9e-aebf-9904422ba430-kube-api-access-98l7w\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.133185 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.133189 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.133459 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.133234 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.133946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.133917 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.135912 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.135892 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.156544 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.156516 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98l7w\" (UniqueName: \"kubernetes.io/projected/9a3de67b-eabc-4b9e-aebf-9904422ba430-kube-api-access-98l7w\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.264200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.264167 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:52:59.565034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:52:59.564959 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx"] Oct 29 23:52:59.568506 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:52:59.568477 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3de67b_eabc_4b9e_aebf_9904422ba430.slice/crio-1efc6bfb44c99f38131834120289575e07ed7a8c012027cb33f29dbf1a011d76 WatchSource:0}: Error finding container 1efc6bfb44c99f38131834120289575e07ed7a8c012027cb33f29dbf1a011d76: Status 404 returned error can't find the container with id 1efc6bfb44c99f38131834120289575e07ed7a8c012027cb33f29dbf1a011d76 Oct 29 23:53:00.061716 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:00.061670 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" event={"ID":"9a3de67b-eabc-4b9e-aebf-9904422ba430","Type":"ContainerStarted","Data":"2537c26f862ba0fe50ab12d9b2b4aec9b19191db41ee536ede2f3655983b1d0b"} Oct 29 23:53:00.061894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:00.061739 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" event={"ID":"9a3de67b-eabc-4b9e-aebf-9904422ba430","Type":"ContainerStarted","Data":"1efc6bfb44c99f38131834120289575e07ed7a8c012027cb33f29dbf1a011d76"} Oct 29 23:53:00.104327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:00.104096 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" podStartSLOduration=2.104074792 podStartE2EDuration="2.104074792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:53:00.101607082 +0000 UTC m=+4177.217526693" watchObservedRunningTime="2026-10-29 23:53:00.104074792 +0000 UTC m=+4177.219994399" Oct 29 23:53:01.071207 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:01.071168 5736 generic.go:356] "Generic (PLEG): container finished" podID="9a3de67b-eabc-4b9e-aebf-9904422ba430" containerID="2537c26f862ba0fe50ab12d9b2b4aec9b19191db41ee536ede2f3655983b1d0b" exitCode=0 Oct 29 23:53:01.071681 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:01.071259 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" event={"ID":"9a3de67b-eabc-4b9e-aebf-9904422ba430","Type":"ContainerDied","Data":"2537c26f862ba0fe50ab12d9b2b4aec9b19191db41ee536ede2f3655983b1d0b"} Oct 29 23:53:02.239232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.239202 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:53:02.370155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.370059 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-xks-ca\") pod \"9a3de67b-eabc-4b9e-aebf-9904422ba430\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " Oct 29 23:53:02.370155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.370105 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-openshift-ca\") pod \"9a3de67b-eabc-4b9e-aebf-9904422ba430\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " Oct 29 23:53:02.370407 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.370286 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3de67b-eabc-4b9e-aebf-9904422ba430-kube-api-access-98l7w\" (UniqueName: \"kubernetes.io/projected/9a3de67b-eabc-4b9e-aebf-9904422ba430-kube-api-access-98l7w\") pod \"9a3de67b-eabc-4b9e-aebf-9904422ba430\" (UID: \"9a3de67b-eabc-4b9e-aebf-9904422ba430\") " Oct 29 23:53:02.370508 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.370482 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-openshift-ca" pod "9a3de67b-eabc-4b9e-aebf-9904422ba430" (UID: "9a3de67b-eabc-4b9e-aebf-9904422ba430"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:02.370723 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.370706 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:02.372277 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.372248 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-xks-ca" pod "9a3de67b-eabc-4b9e-aebf-9904422ba430" (UID: "9a3de67b-eabc-4b9e-aebf-9904422ba430"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:53:02.372383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.372322 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3de67b-eabc-4b9e-aebf-9904422ba430-kube-api-access-98l7w" pod "9a3de67b-eabc-4b9e-aebf-9904422ba430" (UID: "9a3de67b-eabc-4b9e-aebf-9904422ba430"). InnerVolumeSpecName "kube-api-access-98l7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:02.472290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.472255 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98l7w\" (UniqueName: \"kubernetes.io/projected/9a3de67b-eabc-4b9e-aebf-9904422ba430-kube-api-access-98l7w\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:02.472290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:02.472288 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3de67b-eabc-4b9e-aebf-9904422ba430-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:03.087242 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.087211 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" Oct 29 23:53:03.087413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.087242 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx" event={"ID":"9a3de67b-eabc-4b9e-aebf-9904422ba430","Type":"ContainerDied","Data":"1efc6bfb44c99f38131834120289575e07ed7a8c012027cb33f29dbf1a011d76"} Oct 29 23:53:03.087413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.087299 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1efc6bfb44c99f38131834120289575e07ed7a8c012027cb33f29dbf1a011d76" Oct 29 23:53:03.631878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.631851 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:53:03.686360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.686264 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-podinfo\") pod \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " Oct 29 23:53:03.686360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.686352 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-certs\") pod \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " Oct 29 23:53:03.686626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.686458 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-kube-api-access-xqxxf\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-kube-api-access-xqxxf\") pod \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " Oct 29 23:53:03.686626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.686555 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-socket\") pod \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " Oct 29 23:53:03.686626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.686617 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-credential-socket\") pod \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " Oct 29 23:53:03.686758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.686713 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istiod-ca-cert\") pod \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " Oct 29 23:53:03.686758 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.686744 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-token\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-token\") pod \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " Oct 29 23:53:03.686835 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.686797 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-envoy\") pod \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " Oct 29 23:53:03.686887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.686865 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-data\") pod \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\" (UID: \"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666\") " Oct 29 23:53:03.687517 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.687483 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-certs" pod "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" (UID: "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:03.688162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.687666 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-data" pod "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" (UID: "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:03.688162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.687876 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istiod-ca-cert" pod "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" (UID: "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:03.688644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.688549 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-credential-socket" pod "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" (UID: "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:03.689263 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.689231 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-socket" pod "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" (UID: "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:03.691100 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.691073 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-podinfo" pod "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" (UID: "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:53:03.695987 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.695952 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-envoy" pod "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" (UID: "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:03.699642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.699601 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-token" pod "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" (UID: "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:03.699977 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.699872 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-kube-api-access-xqxxf" pod "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" (UID: "2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666"). InnerVolumeSpecName "kube-api-access-xqxxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:03.788505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.788470 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:03.788505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.788499 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:03.788505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.788509 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:03.788770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.788517 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:03.788770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.788525 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:03.788770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.788533 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:03.788770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.788541 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:03.788770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.788550 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqxxf\" (UniqueName: \"kubernetes.io/projected/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-kube-api-access-xqxxf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:03.788770 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:03.788560 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:04.096107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.096067 5736 generic.go:356] "Generic (PLEG): container finished" podID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerID="ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb" exitCode=0 Oct 29 23:53:04.096295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.096124 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" Oct 29 23:53:04.096295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.096154 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" event={"ID":"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666","Type":"ContainerDied","Data":"ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb"} Oct 29 23:53:04.096295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.096207 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk" event={"ID":"2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666","Type":"ContainerDied","Data":"4cb59518417031ec465da86fa5eb489a062ba2ef94194a273c9d05db8bd2b7f1"} Oct 29 23:53:04.096295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.096228 5736 scope.go:122] "RemoveContainer" containerID="ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb" Oct 29 23:53:04.107836 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.107818 5736 scope.go:122] "RemoveContainer" containerID="faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680" Oct 29 23:53:04.117266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.117250 5736 scope.go:122] "RemoveContainer" containerID="ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb" Oct 29 23:53:04.117522 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:53:04.117504 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb\": container with ID starting with ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb not found: ID does not exist" containerID="ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb" Oct 29 23:53:04.117565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.117530 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb"} err="failed to get container status \"ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb\": rpc error: code = NotFound desc = could not find container \"ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb\": container with ID starting with ef4252195b13af7f3c6d0183c3d6cd0c56c383bc32673c4fe151dcbeee370dbb not found: ID does not exist" Oct 29 23:53:04.117565 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.117546 5736 scope.go:122] "RemoveContainer" containerID="faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680" Oct 29 23:53:04.117801 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:53:04.117785 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680\": container with ID starting with faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680 not found: ID does not exist" containerID="faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680" Oct 29 23:53:04.117851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.117804 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680"} err="failed to get container status \"faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680\": rpc error: code = NotFound desc = could not find container \"faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680\": container with ID starting with faa868ad0381465a2664fedaf49fd297d22313b707e82efa8262735d52638680 not found: ID does not exist" Oct 29 23:53:04.170201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.170135 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk"] Oct 29 23:53:04.182267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.182230 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-206095-openshift-default-db4f5f7b5-rlvlk"] Oct 29 23:53:04.502434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.502400 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj"] Oct 29 23:53:04.507104 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:04.507069 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-45036b-bgtsj"] Oct 29 23:53:05.002511 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.002470 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" path="/var/lib/kubelet/pods/2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666/volumes" Oct 29 23:53:05.003137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.003106 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f13e3e46-61b1-436e-87c5-1c06250003b1" path="/var/lib/kubelet/pods/f13e3e46-61b1-436e-87c5-1c06250003b1/volumes" Oct 29 23:53:05.338068 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.337980 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw"] Oct 29 23:53:05.338447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.338433 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" Oct 29 23:53:05.338513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.338448 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" Oct 29 23:53:05.338513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.338461 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" Oct 29 23:53:05.338513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.338466 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" Oct 29 23:53:05.338513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.338487 5736 cpu_manager.go:414] "containerMap: removing container" podUID="9a3de67b-eabc-4b9e-aebf-9904422ba430" containerName="revoke-keys" Oct 29 23:53:05.338513 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.338492 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3de67b-eabc-4b9e-aebf-9904422ba430" containerName="revoke-keys" Oct 29 23:53:05.338831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.338606 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" Oct 29 23:53:05.338831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.338615 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4e89eb-8c53-4fcd-ab11-2fe91e3d3666" containerName="istio-proxy" Oct 29 23:53:05.338831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.338626 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3de67b-eabc-4b9e-aebf-9904422ba430" containerName="revoke-keys" Oct 29 23:53:05.343715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.343694 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.363232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.363202 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw"] Oct 29 23:53:05.393846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.393817 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:53:05.396172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.396146 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:53:05.405253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.405227 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.405371 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.405346 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.405424 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.405380 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbqzk\" (UniqueName: \"kubernetes.io/projected/8ab539a6-d3a9-4314-9140-9efa4f4521e7-kube-api-access-kbqzk\") pod \"maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.506898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.506872 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.507041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.506934 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.507041 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.506960 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbqzk\" (UniqueName: \"kubernetes.io/projected/8ab539a6-d3a9-4314-9140-9efa4f4521e7-kube-api-access-kbqzk\") pod \"maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.507515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.507482 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.509264 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.509244 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.529983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.529953 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbqzk\" (UniqueName: \"kubernetes.io/projected/8ab539a6-d3a9-4314-9140-9efa4f4521e7-kube-api-access-kbqzk\") pod \"maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.657254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.657179 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:05.960421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:05.960388 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw"] Oct 29 23:53:05.964166 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:53:05.964130 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab539a6_d3a9_4314_9140_9efa4f4521e7.slice/crio-dabd1ff47db6bfd526820eafcbac18606a8b014ec0e4d512a92fa834ba300b1a WatchSource:0}: Error finding container dabd1ff47db6bfd526820eafcbac18606a8b014ec0e4d512a92fa834ba300b1a: Status 404 returned error can't find the container with id dabd1ff47db6bfd526820eafcbac18606a8b014ec0e4d512a92fa834ba300b1a Oct 29 23:53:06.114166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:06.114131 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" event={"ID":"8ab539a6-d3a9-4314-9140-9efa4f4521e7","Type":"ContainerStarted","Data":"dabd1ff47db6bfd526820eafcbac18606a8b014ec0e4d512a92fa834ba300b1a"} Oct 29 23:53:07.122831 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:07.122741 5736 generic.go:356] "Generic (PLEG): container finished" podID="8ab539a6-d3a9-4314-9140-9efa4f4521e7" containerID="d80d4b5302cb8282c7c8e2d17f5bfaba5ac81986daacf6d686af313210413b30" exitCode=0 Oct 29 23:53:07.123188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:07.122829 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" event={"ID":"8ab539a6-d3a9-4314-9140-9efa4f4521e7","Type":"ContainerDied","Data":"d80d4b5302cb8282c7c8e2d17f5bfaba5ac81986daacf6d686af313210413b30"} Oct 29 23:53:08.268003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.267974 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:08.335944 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.335906 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-openshift-ca\") pod \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " Oct 29 23:53:08.336319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.336297 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab539a6-d3a9-4314-9140-9efa4f4521e7-kube-api-access-kbqzk\" (UniqueName: \"kubernetes.io/projected/8ab539a6-d3a9-4314-9140-9efa4f4521e7-kube-api-access-kbqzk\") pod \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " Oct 29 23:53:08.336566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.336548 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-xks-ca\") pod \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\" (UID: \"8ab539a6-d3a9-4314-9140-9efa4f4521e7\") " Oct 29 23:53:08.336817 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.336786 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-openshift-ca" pod "8ab539a6-d3a9-4314-9140-9efa4f4521e7" (UID: "8ab539a6-d3a9-4314-9140-9efa4f4521e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:08.337519 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.337493 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:08.341491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.341464 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-xks-ca" pod "8ab539a6-d3a9-4314-9140-9efa4f4521e7" (UID: "8ab539a6-d3a9-4314-9140-9efa4f4521e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:53:08.341644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.341496 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab539a6-d3a9-4314-9140-9efa4f4521e7-kube-api-access-kbqzk" pod "8ab539a6-d3a9-4314-9140-9efa4f4521e7" (UID: "8ab539a6-d3a9-4314-9140-9efa4f4521e7"). InnerVolumeSpecName "kube-api-access-kbqzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:08.438315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.438243 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbqzk\" (UniqueName: \"kubernetes.io/projected/8ab539a6-d3a9-4314-9140-9efa4f4521e7-kube-api-access-kbqzk\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:08.438315 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:08.438273 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab539a6-d3a9-4314-9140-9efa4f4521e7-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:09.139898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:09.139860 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" event={"ID":"8ab539a6-d3a9-4314-9140-9efa4f4521e7","Type":"ContainerDied","Data":"dabd1ff47db6bfd526820eafcbac18606a8b014ec0e4d512a92fa834ba300b1a"} Oct 29 23:53:09.139898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:09.139892 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw" Oct 29 23:53:09.140107 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:09.139920 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dabd1ff47db6bfd526820eafcbac18606a8b014ec0e4d512a92fa834ba300b1a" Oct 29 23:53:09.360531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:09.360448 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl"] Oct 29 23:53:14.360750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:14.360682 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" podUID="c17d87bb-02f0-4574-9f71-b6c97321b347" containerName="payload-processing" containerID="cri-o://afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192" gracePeriod=25 Oct 29 23:53:14.620211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:14.620144 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:53:14.809373 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:14.809343 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17d87bb-02f0-4574-9f71-b6c97321b347-kube-api-access-67sxt\" (UniqueName: \"kubernetes.io/projected/c17d87bb-02f0-4574-9f71-b6c97321b347-kube-api-access-67sxt\") pod \"c17d87bb-02f0-4574-9f71-b6c97321b347\" (UID: \"c17d87bb-02f0-4574-9f71-b6c97321b347\") " Oct 29 23:53:14.809554 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:14.809392 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17d87bb-02f0-4574-9f71-b6c97321b347-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17d87bb-02f0-4574-9f71-b6c97321b347-plugins-config-volume\") pod \"c17d87bb-02f0-4574-9f71-b6c97321b347\" (UID: \"c17d87bb-02f0-4574-9f71-b6c97321b347\") " Oct 29 23:53:14.809876 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:14.809839 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17d87bb-02f0-4574-9f71-b6c97321b347-plugins-config-volume" pod "c17d87bb-02f0-4574-9f71-b6c97321b347" (UID: "c17d87bb-02f0-4574-9f71-b6c97321b347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:14.811451 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:14.811425 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17d87bb-02f0-4574-9f71-b6c97321b347-kube-api-access-67sxt" pod "c17d87bb-02f0-4574-9f71-b6c97321b347" (UID: "c17d87bb-02f0-4574-9f71-b6c97321b347"). InnerVolumeSpecName "kube-api-access-67sxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:14.911078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:14.911018 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67sxt\" (UniqueName: \"kubernetes.io/projected/c17d87bb-02f0-4574-9f71-b6c97321b347-kube-api-access-67sxt\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:14.911078 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:14.911045 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17d87bb-02f0-4574-9f71-b6c97321b347-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:15.193841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.193741 5736 generic.go:356] "Generic (PLEG): container finished" podID="c17d87bb-02f0-4574-9f71-b6c97321b347" containerID="afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192" exitCode=0 Oct 29 23:53:15.193841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.193787 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" Oct 29 23:53:15.193841 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.193822 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" event={"ID":"c17d87bb-02f0-4574-9f71-b6c97321b347","Type":"ContainerDied","Data":"afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192"} Oct 29 23:53:15.194166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.193886 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl" event={"ID":"c17d87bb-02f0-4574-9f71-b6c97321b347","Type":"ContainerDied","Data":"34f94e9cd4595f25ba429b996ea1163ff05c7a8766dfd4459df5ed8bd70866f9"} Oct 29 23:53:15.194166 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.193918 5736 scope.go:122] "RemoveContainer" containerID="afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192" Oct 29 23:53:15.205033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.205013 5736 scope.go:122] "RemoveContainer" containerID="afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192" Oct 29 23:53:15.205325 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:53:15.205305 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192\": container with ID starting with afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192 not found: ID does not exist" containerID="afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192" Oct 29 23:53:15.205386 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.205333 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192"} err="failed to get container status \"afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192\": rpc error: code = NotFound desc = could not find container \"afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192\": container with ID starting with afd3eb9e09d673c8194a7ff31f44b1738bd28ebae9466c4e7049d49b3f282192 not found: ID does not exist" Oct 29 23:53:15.256137 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.256100 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl"] Oct 29 23:53:15.268781 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.268741 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-a4f2db-858c657d4d-gpxxl"] Oct 29 23:53:15.837061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.837018 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx"] Oct 29 23:53:15.841150 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:15.841119 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-a4f2db-82qnx"] Oct 29 23:53:17.002928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:17.002883 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a3de67b-eabc-4b9e-aebf-9904422ba430" path="/var/lib/kubelet/pods/9a3de67b-eabc-4b9e-aebf-9904422ba430/volumes" Oct 29 23:53:17.003498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:17.003475 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17d87bb-02f0-4574-9f71-b6c97321b347" path="/var/lib/kubelet/pods/c17d87bb-02f0-4574-9f71-b6c97321b347/volumes" Oct 29 23:53:21.838370 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.838331 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v"] Oct 29 23:53:21.838854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.838838 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8ab539a6-d3a9-4314-9140-9efa4f4521e7" containerName="revoke-keys" Oct 29 23:53:21.838905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.838854 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab539a6-d3a9-4314-9140-9efa4f4521e7" containerName="revoke-keys" Oct 29 23:53:21.838905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.838894 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c17d87bb-02f0-4574-9f71-b6c97321b347" containerName="payload-processing" Oct 29 23:53:21.838905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.838900 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17d87bb-02f0-4574-9f71-b6c97321b347" containerName="payload-processing" Oct 29 23:53:21.839030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.838988 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17d87bb-02f0-4574-9f71-b6c97321b347" containerName="payload-processing" Oct 29 23:53:21.839030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.839003 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab539a6-d3a9-4314-9140-9efa4f4521e7" containerName="revoke-keys" Oct 29 23:53:21.843771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.843748 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:21.861520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.861499 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:53:21.873913 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.873888 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:53:21.876497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.876471 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v"] Oct 29 23:53:21.885875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.885844 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:21.886025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.886006 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz578\" (UniqueName: \"kubernetes.io/projected/fc9e0d6a-c063-414a-b687-4a545d35a11c-kube-api-access-fz578\") pod \"maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:21.886086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.886055 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:21.987106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.987078 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz578\" (UniqueName: \"kubernetes.io/projected/fc9e0d6a-c063-414a-b687-4a545d35a11c-kube-api-access-fz578\") pod \"maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:21.987210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.987133 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:21.987210 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.987185 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:21.987927 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.987896 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:21.989559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:21.989532 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:22.009997 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:22.009971 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz578\" (UniqueName: \"kubernetes.io/projected/fc9e0d6a-c063-414a-b687-4a545d35a11c-kube-api-access-fz578\") pod \"maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:22.157314 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:22.157239 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:22.461358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:22.461325 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v"] Oct 29 23:53:22.464751 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:53:22.464723 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9e0d6a_c063_414a_b687_4a545d35a11c.slice/crio-00b8d88dec7d8c819e2b7db6eaca7838225e5df845e95667ba61b6cef2ac5c91 WatchSource:0}: Error finding container 00b8d88dec7d8c819e2b7db6eaca7838225e5df845e95667ba61b6cef2ac5c91: Status 404 returned error can't find the container with id 00b8d88dec7d8c819e2b7db6eaca7838225e5df845e95667ba61b6cef2ac5c91 Oct 29 23:53:23.268779 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:23.268732 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" event={"ID":"fc9e0d6a-c063-414a-b687-4a545d35a11c","Type":"ContainerStarted","Data":"75e6ac18bebc19427cc09a7dc27f2009733511603e00957b86996cec47806535"} Oct 29 23:53:23.269384 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:23.269355 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" event={"ID":"fc9e0d6a-c063-414a-b687-4a545d35a11c","Type":"ContainerStarted","Data":"00b8d88dec7d8c819e2b7db6eaca7838225e5df845e95667ba61b6cef2ac5c91"} Oct 29 23:53:23.313936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:23.313871 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" podStartSLOduration=2.313856252 podStartE2EDuration="2.313856252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:53:23.31105368 +0000 UTC m=+4200.426973287" watchObservedRunningTime="2026-10-29 23:53:23.313856252 +0000 UTC m=+4200.429775858" Oct 29 23:53:23.776756 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:23.776709 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw"] Oct 29 23:53:23.779362 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:23.779325 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a84df8-nktbw"] Oct 29 23:53:24.211231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:24.211198 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:53:24.273471 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:24.273443 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vppzr_d839fc45-f2f4-4ce4-95d8-9e3145342973/ovn-acl-logging/0.log" Oct 29 23:53:24.277184 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:24.277144 5736 generic.go:356] "Generic (PLEG): container finished" podID="fc9e0d6a-c063-414a-b687-4a545d35a11c" containerID="75e6ac18bebc19427cc09a7dc27f2009733511603e00957b86996cec47806535" exitCode=0 Oct 29 23:53:24.277350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:24.277227 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" event={"ID":"fc9e0d6a-c063-414a-b687-4a545d35a11c","Type":"ContainerDied","Data":"75e6ac18bebc19427cc09a7dc27f2009733511603e00957b86996cec47806535"} Oct 29 23:53:25.003599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.003556 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab539a6-d3a9-4314-9140-9efa4f4521e7" path="/var/lib/kubelet/pods/8ab539a6-d3a9-4314-9140-9efa4f4521e7/volumes" Oct 29 23:53:25.440528 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.440497 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:25.543398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.543321 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-xks-ca\") pod \"fc9e0d6a-c063-414a-b687-4a545d35a11c\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " Oct 29 23:53:25.543542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.543446 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-openshift-ca\") pod \"fc9e0d6a-c063-414a-b687-4a545d35a11c\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " Oct 29 23:53:25.543542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.543483 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc9e0d6a-c063-414a-b687-4a545d35a11c-kube-api-access-fz578\" (UniqueName: \"kubernetes.io/projected/fc9e0d6a-c063-414a-b687-4a545d35a11c-kube-api-access-fz578\") pod \"fc9e0d6a-c063-414a-b687-4a545d35a11c\" (UID: \"fc9e0d6a-c063-414a-b687-4a545d35a11c\") " Oct 29 23:53:25.543873 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.543836 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-openshift-ca" pod "fc9e0d6a-c063-414a-b687-4a545d35a11c" (UID: "fc9e0d6a-c063-414a-b687-4a545d35a11c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:25.545452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.545426 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-xks-ca" pod "fc9e0d6a-c063-414a-b687-4a545d35a11c" (UID: "fc9e0d6a-c063-414a-b687-4a545d35a11c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:53:25.545629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.545604 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9e0d6a-c063-414a-b687-4a545d35a11c-kube-api-access-fz578" pod "fc9e0d6a-c063-414a-b687-4a545d35a11c" (UID: "fc9e0d6a-c063-414a-b687-4a545d35a11c"). InnerVolumeSpecName "kube-api-access-fz578". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:25.644548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.644521 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:25.644548 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.644547 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9e0d6a-c063-414a-b687-4a545d35a11c-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:25.644727 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:25.644558 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz578\" (UniqueName: \"kubernetes.io/projected/fc9e0d6a-c063-414a-b687-4a545d35a11c-kube-api-access-fz578\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:26.292181 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:26.292143 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" Oct 29 23:53:26.292360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:26.292186 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v" event={"ID":"fc9e0d6a-c063-414a-b687-4a545d35a11c","Type":"ContainerDied","Data":"00b8d88dec7d8c819e2b7db6eaca7838225e5df845e95667ba61b6cef2ac5c91"} Oct 29 23:53:26.292360 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:26.292244 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00b8d88dec7d8c819e2b7db6eaca7838225e5df845e95667ba61b6cef2ac5c91" Oct 29 23:53:26.824715 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:53:26.824676 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:53:31.413174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.413124 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp"] Oct 29 23:53:31.413837 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.413814 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fc9e0d6a-c063-414a-b687-4a545d35a11c" containerName="revoke-keys" Oct 29 23:53:31.413926 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.413839 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc9e0d6a-c063-414a-b687-4a545d35a11c" containerName="revoke-keys" Oct 29 23:53:31.414033 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.414018 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc9e0d6a-c063-414a-b687-4a545d35a11c" containerName="revoke-keys" Oct 29 23:53:31.417304 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.417281 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.438760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.438733 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp"] Oct 29 23:53:31.460197 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.460161 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:53:31.460323 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.460243 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:53:31.508414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.508389 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clpxq\" (UniqueName: \"kubernetes.io/projected/a7d8dc94-b54c-4179-8011-02855b6b865a-kube-api-access-clpxq\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.508559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.508423 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.508559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.508528 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.609632 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.609574 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.609771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.609753 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clpxq\" (UniqueName: \"kubernetes.io/projected/a7d8dc94-b54c-4179-8011-02855b6b865a-kube-api-access-clpxq\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.609847 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.609789 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.610299 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.610275 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.612275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.612251 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.632934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.632914 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clpxq\" (UniqueName: \"kubernetes.io/projected/a7d8dc94-b54c-4179-8011-02855b6b865a-kube-api-access-clpxq\") pod \"maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:31.654735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.654707 5736 scope.go:122] "RemoveContainer" containerID="5ee3281c41ac411ee6ac2c7964f1393b80d888d38c4ec7c2474ad9f325cf9a77" Oct 29 23:53:31.676561 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.676537 5736 scope.go:122] "RemoveContainer" containerID="ad1475236eeddd66570c2442f42ac8eb57763029ba50af86a5cacbef3274e14a" Oct 29 23:53:31.686198 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.686179 5736 scope.go:122] "RemoveContainer" containerID="fc2b6f959da4de0aff0b3cfe7a810cbde1a48939bb2eab82f27a180adb7c6334" Oct 29 23:53:31.695147 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.695128 5736 scope.go:122] "RemoveContainer" containerID="8214f45212d049926b1464d56e8c008abe0524af863559a0d036e17ded0484e4" Oct 29 23:53:31.703902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.703885 5736 scope.go:122] "RemoveContainer" containerID="5df5d90dd2a6f6979736dc62df07c20bbc1a7d410cd7582380aecfe105f12d06" Oct 29 23:53:31.729878 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:31.729858 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:32.044317 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:32.044281 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp"] Oct 29 23:53:32.048783 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:53:32.048754 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d8dc94_b54c_4179_8011_02855b6b865a.slice/crio-42a926d4b6798980f0b207a2a8fa2e5b82bbcbeb4a81c5eb0d4a49ed4ea007d8 WatchSource:0}: Error finding container 42a926d4b6798980f0b207a2a8fa2e5b82bbcbeb4a81c5eb0d4a49ed4ea007d8: Status 404 returned error can't find the container with id 42a926d4b6798980f0b207a2a8fa2e5b82bbcbeb4a81c5eb0d4a49ed4ea007d8 Oct 29 23:53:32.238171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:32.238139 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b"] Oct 29 23:53:32.270598 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:32.270546 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz"] Oct 29 23:53:32.355960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:32.355924 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" event={"ID":"a7d8dc94-b54c-4179-8011-02855b6b865a","Type":"ContainerStarted","Data":"c7140beab5e36607c69111458e59d9703d87a7b1e49118d36729b6e97eaf064e"} Oct 29 23:53:32.356119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:32.355989 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" event={"ID":"a7d8dc94-b54c-4179-8011-02855b6b865a","Type":"ContainerStarted","Data":"42a926d4b6798980f0b207a2a8fa2e5b82bbcbeb4a81c5eb0d4a49ed4ea007d8"} Oct 29 23:53:32.397497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:32.397439 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" podStartSLOduration=1.397420002 podStartE2EDuration="1.397420002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:53:32.395926412 +0000 UTC m=+4209.511846019" watchObservedRunningTime="2026-10-29 23:53:32.397420002 +0000 UTC m=+4209.513339610" Oct 29 23:53:33.365896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:33.365856 5736 generic.go:356] "Generic (PLEG): container finished" podID="a7d8dc94-b54c-4179-8011-02855b6b865a" containerID="c7140beab5e36607c69111458e59d9703d87a7b1e49118d36729b6e97eaf064e" exitCode=0 Oct 29 23:53:33.366300 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:33.365940 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" event={"ID":"a7d8dc94-b54c-4179-8011-02855b6b865a","Type":"ContainerDied","Data":"c7140beab5e36607c69111458e59d9703d87a7b1e49118d36729b6e97eaf064e"} Oct 29 23:53:34.546113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.546086 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:34.645327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.645251 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d8dc94-b54c-4179-8011-02855b6b865a-kube-api-access-clpxq\" (UniqueName: \"kubernetes.io/projected/a7d8dc94-b54c-4179-8011-02855b6b865a-kube-api-access-clpxq\") pod \"a7d8dc94-b54c-4179-8011-02855b6b865a\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " Oct 29 23:53:34.645327 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.645301 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-xks-ca\") pod \"a7d8dc94-b54c-4179-8011-02855b6b865a\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " Oct 29 23:53:34.645597 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.645335 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-openshift-ca\") pod \"a7d8dc94-b54c-4179-8011-02855b6b865a\" (UID: \"a7d8dc94-b54c-4179-8011-02855b6b865a\") " Oct 29 23:53:34.645846 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.645816 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-openshift-ca" pod "a7d8dc94-b54c-4179-8011-02855b6b865a" (UID: "a7d8dc94-b54c-4179-8011-02855b6b865a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:34.647412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.647386 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-xks-ca" pod "a7d8dc94-b54c-4179-8011-02855b6b865a" (UID: "a7d8dc94-b54c-4179-8011-02855b6b865a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:53:34.647515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.647449 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d8dc94-b54c-4179-8011-02855b6b865a-kube-api-access-clpxq" pod "a7d8dc94-b54c-4179-8011-02855b6b865a" (UID: "a7d8dc94-b54c-4179-8011-02855b6b865a"). InnerVolumeSpecName "kube-api-access-clpxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:34.746650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.746609 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clpxq\" (UniqueName: \"kubernetes.io/projected/a7d8dc94-b54c-4179-8011-02855b6b865a-kube-api-access-clpxq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:34.746650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.746650 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:34.746855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:34.746668 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8dc94-b54c-4179-8011-02855b6b865a-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:35.381489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:35.381396 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" event={"ID":"a7d8dc94-b54c-4179-8011-02855b6b865a","Type":"ContainerDied","Data":"42a926d4b6798980f0b207a2a8fa2e5b82bbcbeb4a81c5eb0d4a49ed4ea007d8"} Oct 29 23:53:35.381489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:35.381452 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42a926d4b6798980f0b207a2a8fa2e5b82bbcbeb4a81c5eb0d4a49ed4ea007d8" Oct 29 23:53:35.381489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:35.381412 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp" Oct 29 23:53:37.239266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.239223 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" podUID="1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78" containerName="payload-processing" containerID="cri-o://12566b9ac42e864e3109359b5fe5e844cb09e3f98342263afae27f8b15cbd364" gracePeriod=25 Oct 29 23:53:37.271455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.271400 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" podUID="fdf7b783-43f7-49aa-8594-8a328ece36e0" containerName="payload-pre-processing" containerID="cri-o://82711292a21f775b4e056c1c854d70ce66fd8e23d53ec64a4c43cf9307044d25" gracePeriod=25 Oct 29 23:53:37.398441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.398404 5736 generic.go:356] "Generic (PLEG): container finished" podID="fdf7b783-43f7-49aa-8594-8a328ece36e0" containerID="82711292a21f775b4e056c1c854d70ce66fd8e23d53ec64a4c43cf9307044d25" exitCode=0 Oct 29 23:53:37.398653 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.398491 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" event={"ID":"fdf7b783-43f7-49aa-8594-8a328ece36e0","Type":"ContainerDied","Data":"82711292a21f775b4e056c1c854d70ce66fd8e23d53ec64a4c43cf9307044d25"} Oct 29 23:53:37.400887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.400856 5736 generic.go:356] "Generic (PLEG): container finished" podID="1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78" containerID="12566b9ac42e864e3109359b5fe5e844cb09e3f98342263afae27f8b15cbd364" exitCode=0 Oct 29 23:53:37.401065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.400941 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" event={"ID":"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78","Type":"ContainerDied","Data":"12566b9ac42e864e3109359b5fe5e844cb09e3f98342263afae27f8b15cbd364"} Oct 29 23:53:37.532738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.532711 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:53:37.568069 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.568040 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:53:37.678627 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.678595 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf7b783-43f7-49aa-8594-8a328ece36e0-kube-api-access-7h4r9\" (UniqueName: \"kubernetes.io/projected/fdf7b783-43f7-49aa-8594-8a328ece36e0-kube-api-access-7h4r9\") pod \"fdf7b783-43f7-49aa-8594-8a328ece36e0\" (UID: \"fdf7b783-43f7-49aa-8594-8a328ece36e0\") " Oct 29 23:53:37.678825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.678635 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf7b783-43f7-49aa-8594-8a328ece36e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf7b783-43f7-49aa-8594-8a328ece36e0-plugins-config-volume\") pod \"fdf7b783-43f7-49aa-8594-8a328ece36e0\" (UID: \"fdf7b783-43f7-49aa-8594-8a328ece36e0\") " Oct 29 23:53:37.678825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.678717 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-plugins-config-volume\") pod \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\" (UID: \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\") " Oct 29 23:53:37.678965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.678835 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-kube-api-access-d8nz6\" (UniqueName: \"kubernetes.io/projected/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-kube-api-access-d8nz6\") pod \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\" (UID: \"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78\") " Oct 29 23:53:37.679086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.679057 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf7b783-43f7-49aa-8594-8a328ece36e0-plugins-config-volume" pod "fdf7b783-43f7-49aa-8594-8a328ece36e0" (UID: "fdf7b783-43f7-49aa-8594-8a328ece36e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:37.679171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.679128 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-plugins-config-volume" pod "1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78" (UID: "1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:37.679230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.679208 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf7b783-43f7-49aa-8594-8a328ece36e0-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:37.679296 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.679230 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:37.680955 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.680930 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf7b783-43f7-49aa-8594-8a328ece36e0-kube-api-access-7h4r9" pod "fdf7b783-43f7-49aa-8594-8a328ece36e0" (UID: "fdf7b783-43f7-49aa-8594-8a328ece36e0"). InnerVolumeSpecName "kube-api-access-7h4r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:37.681063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.680960 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-kube-api-access-d8nz6" pod "1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78" (UID: "1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78"). InnerVolumeSpecName "kube-api-access-d8nz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:37.780188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.780118 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8nz6\" (UniqueName: \"kubernetes.io/projected/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78-kube-api-access-d8nz6\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:37.780188 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:37.780148 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h4r9\" (UniqueName: \"kubernetes.io/projected/fdf7b783-43f7-49aa-8594-8a328ece36e0-kube-api-access-7h4r9\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:38.409966 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.409933 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" Oct 29 23:53:38.410421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.409965 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz" event={"ID":"fdf7b783-43f7-49aa-8594-8a328ece36e0","Type":"ContainerDied","Data":"6054c17e86f07a9de954b1f45e53bc9419580018c76cf0b706c8fb4f0a0acadb"} Oct 29 23:53:38.410421 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.410033 5736 scope.go:122] "RemoveContainer" containerID="82711292a21f775b4e056c1c854d70ce66fd8e23d53ec64a4c43cf9307044d25" Oct 29 23:53:38.411906 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.411878 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" Oct 29 23:53:38.412008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.411916 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b" event={"ID":"1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78","Type":"ContainerDied","Data":"f438bd9cc188d49a2c21b572b0bf651d60257547dedd7a120a2129255f2dad0e"} Oct 29 23:53:38.422094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.422076 5736 scope.go:122] "RemoveContainer" containerID="12566b9ac42e864e3109359b5fe5e844cb09e3f98342263afae27f8b15cbd364" Oct 29 23:53:38.461389 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.461364 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz"] Oct 29 23:53:38.473004 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.472982 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e59cd-6c9968f899-7mnfz"] Oct 29 23:53:38.497806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.497761 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b"] Oct 29 23:53:38.502294 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:38.502250 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e59cd-78fdbc968b-8xs7b"] Oct 29 23:53:39.002477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:39.002439 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78" path="/var/lib/kubelet/pods/1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78/volumes" Oct 29 23:53:39.002888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:39.002872 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf7b783-43f7-49aa-8594-8a328ece36e0" path="/var/lib/kubelet/pods/fdf7b783-43f7-49aa-8594-8a328ece36e0/volumes" Oct 29 23:53:41.830437 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:41.830401 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j"] Oct 29 23:53:41.859342 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:41.859311 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9"] Oct 29 23:53:44.612191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:44.612156 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82"] Oct 29 23:53:44.612663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:44.612344 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" podUID="6e1f74d3-9981-4244-a824-ee910022a624" containerName="maas-api" containerID="cri-o://6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48" gracePeriod=30 Oct 29 23:53:44.913731 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:44.913645 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:53:45.059297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.059257 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-maas-api-tls\") pod \"6e1f74d3-9981-4244-a824-ee910022a624\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " Oct 29 23:53:45.059475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.059418 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1f74d3-9981-4244-a824-ee910022a624-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f74d3-9981-4244-a824-ee910022a624-openshift-service-ca\") pod \"6e1f74d3-9981-4244-a824-ee910022a624\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " Oct 29 23:53:45.059542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.059494 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-metrics-certs\") pod \"6e1f74d3-9981-4244-a824-ee910022a624\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " Oct 29 23:53:45.059644 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.059540 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1f74d3-9981-4244-a824-ee910022a624-kube-api-access-zk96v\" (UniqueName: \"kubernetes.io/projected/6e1f74d3-9981-4244-a824-ee910022a624-kube-api-access-zk96v\") pod \"6e1f74d3-9981-4244-a824-ee910022a624\" (UID: \"6e1f74d3-9981-4244-a824-ee910022a624\") " Oct 29 23:53:45.059833 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.059808 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1f74d3-9981-4244-a824-ee910022a624-openshift-service-ca" pod "6e1f74d3-9981-4244-a824-ee910022a624" (UID: "6e1f74d3-9981-4244-a824-ee910022a624"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:45.059999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.059981 5736 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1f74d3-9981-4244-a824-ee910022a624-openshift-service-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:45.061492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.061464 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-maas-api-tls" pod "6e1f74d3-9981-4244-a824-ee910022a624" (UID: "6e1f74d3-9981-4244-a824-ee910022a624"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:53:45.061663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.061538 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-metrics-certs" pod "6e1f74d3-9981-4244-a824-ee910022a624" (UID: "6e1f74d3-9981-4244-a824-ee910022a624"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:53:45.061717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.061670 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1f74d3-9981-4244-a824-ee910022a624-kube-api-access-zk96v" pod "6e1f74d3-9981-4244-a824-ee910022a624" (UID: "6e1f74d3-9981-4244-a824-ee910022a624"). InnerVolumeSpecName "kube-api-access-zk96v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:45.161497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.161459 5736 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-metrics-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:45.161497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.161491 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk96v\" (UniqueName: \"kubernetes.io/projected/6e1f74d3-9981-4244-a824-ee910022a624-kube-api-access-zk96v\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:45.161497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.161500 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1f74d3-9981-4244-a824-ee910022a624-maas-api-tls\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:45.462467 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.462426 5736 generic.go:356] "Generic (PLEG): container finished" podID="6e1f74d3-9981-4244-a824-ee910022a624" containerID="6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48" exitCode=0 Oct 29 23:53:45.462805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.462468 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" Oct 29 23:53:45.462805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.462508 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" event={"ID":"6e1f74d3-9981-4244-a824-ee910022a624","Type":"ContainerDied","Data":"6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48"} Oct 29 23:53:45.462805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.462563 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82" event={"ID":"6e1f74d3-9981-4244-a824-ee910022a624","Type":"ContainerDied","Data":"ce1630fb88305498b30d2db3abf2e853ebdc61d415429b707a945c52ce9be2f2"} Oct 29 23:53:45.462805 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.462602 5736 scope.go:122] "RemoveContainer" containerID="6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48" Oct 29 23:53:45.474168 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.474152 5736 scope.go:122] "RemoveContainer" containerID="6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48" Oct 29 23:53:45.474427 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:53:45.474407 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48\": container with ID starting with 6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48 not found: ID does not exist" containerID="6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48" Oct 29 23:53:45.474499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.474435 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48"} err="failed to get container status \"6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48\": rpc error: code = NotFound desc = could not find container \"6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48\": container with ID starting with 6c2d148940ee850d93357f919fad174e7cb8ef33ae46384806cc026b49b58a48 not found: ID does not exist" Oct 29 23:53:45.481173 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.481143 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v"] Oct 29 23:53:45.483626 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.483602 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0e59cd-vsw8v"] Oct 29 23:53:45.528470 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.527373 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82"] Oct 29 23:53:45.529881 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:45.529850 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e59cd-67c4cbf5f5-gsg82"] Oct 29 23:53:46.831444 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:46.831394 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" podUID="ca615e7b-9e24-4aa4-a005-bbd24c17a39a" containerName="payload-processing" containerID="cri-o://9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49" gracePeriod=25 Oct 29 23:53:46.859978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:46.859944 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" podUID="a7efa29c-fb60-46e2-baa2-15d5c816a4e7" containerName="payload-pre-processing" containerID="cri-o://9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4" gracePeriod=25 Oct 29 23:53:47.003040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:47.002993 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1f74d3-9981-4244-a824-ee910022a624" path="/var/lib/kubelet/pods/6e1f74d3-9981-4244-a824-ee910022a624/volumes" Oct 29 23:53:47.003541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:47.003517 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc9e0d6a-c063-414a-b687-4a545d35a11c" path="/var/lib/kubelet/pods/fc9e0d6a-c063-414a-b687-4a545d35a11c/volumes" Oct 29 23:53:48.256337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:48.256298 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf"] Oct 29 23:53:48.256775 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:48.256591 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" containerID="cri-o://82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533" gracePeriod=30 Oct 29 23:53:50.125658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:50.125616 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" podUID="a7efa29c-fb60-46e2-baa2-15d5c816a4e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:53:50.125658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:50.125640 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" podUID="ca615e7b-9e24-4aa4-a005-bbd24c17a39a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:53:50.483640 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:50.483603 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484"] Oct 29 23:53:50.495518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:50.495484 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c"] Oct 29 23:53:50.502965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:50.502935 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw"] Oct 29 23:53:50.962965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:50.962933 5736 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:53:50.963141 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:50.962983 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:53:53.505609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.505570 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:53:53.526603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.526524 5736 generic.go:356] "Generic (PLEG): container finished" podID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerID="82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533" exitCode=0 Oct 29 23:53:53.526603 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.526568 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" Oct 29 23:53:53.526784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.526623 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" event={"ID":"4a7924e4-0728-45ba-9f05-1a4e1d725062","Type":"ContainerDied","Data":"82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533"} Oct 29 23:53:53.526784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.526680 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf" event={"ID":"4a7924e4-0728-45ba-9f05-1a4e1d725062","Type":"ContainerDied","Data":"b68f6ce4618544363fc3e78a2c97635669a329e514d411074169a62ca47c35d7"} Oct 29 23:53:53.526784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.526703 5736 scope.go:122] "RemoveContainer" containerID="82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533" Oct 29 23:53:53.536892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.536861 5736 scope.go:122] "RemoveContainer" containerID="82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533" Oct 29 23:53:53.537186 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:53:53.537164 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533\": container with ID starting with 82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533 not found: ID does not exist" containerID="82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533" Oct 29 23:53:53.537282 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.537196 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533"} err="failed to get container status \"82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533\": rpc error: code = NotFound desc = could not find container \"82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533\": container with ID starting with 82bc18f36d26e89a3a4d6b6b1871baf372b6e34654451e7bf2194b5635047533 not found: ID does not exist" Oct 29 23:53:53.545642 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545619 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-credential-socket\") pod \"4a7924e4-0728-45ba-9f05-1a4e1d725062\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " Oct 29 23:53:53.545760 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545730 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-socket\") pod \"4a7924e4-0728-45ba-9f05-1a4e1d725062\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " Oct 29 23:53:53.545830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545770 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a7924e4-0728-45ba-9f05-1a4e1d725062-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7924e4-0728-45ba-9f05-1a4e1d725062-istiod-ca-cert\") pod \"4a7924e4-0728-45ba-9f05-1a4e1d725062\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " Oct 29 23:53:53.545830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545810 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-data\") pod \"4a7924e4-0728-45ba-9f05-1a4e1d725062\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " Oct 29 23:53:53.545949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545847 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-credential-socket" pod "4a7924e4-0728-45ba-9f05-1a4e1d725062" (UID: "4a7924e4-0728-45ba-9f05-1a4e1d725062"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:53.545949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545865 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-token\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-token\") pod \"4a7924e4-0728-45ba-9f05-1a4e1d725062\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " Oct 29 23:53:53.545949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545911 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-envoy\") pod \"4a7924e4-0728-45ba-9f05-1a4e1d725062\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " Oct 29 23:53:53.545949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545943 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-certs\") pod \"4a7924e4-0728-45ba-9f05-1a4e1d725062\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " Oct 29 23:53:53.546175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545989 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-kube-api-access-6wbq7\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-kube-api-access-6wbq7\") pod \"4a7924e4-0728-45ba-9f05-1a4e1d725062\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " Oct 29 23:53:53.546175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.545994 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-socket" pod "4a7924e4-0728-45ba-9f05-1a4e1d725062" (UID: "4a7924e4-0728-45ba-9f05-1a4e1d725062"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:53.546175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.546040 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-podinfo\") pod \"4a7924e4-0728-45ba-9f05-1a4e1d725062\" (UID: \"4a7924e4-0728-45ba-9f05-1a4e1d725062\") " Oct 29 23:53:53.546175 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.546072 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-data" pod "4a7924e4-0728-45ba-9f05-1a4e1d725062" (UID: "4a7924e4-0728-45ba-9f05-1a4e1d725062"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:53.546379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.546246 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-certs" pod "4a7924e4-0728-45ba-9f05-1a4e1d725062" (UID: "4a7924e4-0728-45ba-9f05-1a4e1d725062"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:53.546379 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.546258 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7924e4-0728-45ba-9f05-1a4e1d725062-istiod-ca-cert" pod "4a7924e4-0728-45ba-9f05-1a4e1d725062" (UID: "4a7924e4-0728-45ba-9f05-1a4e1d725062"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:53.546537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.546514 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:53.546620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.546545 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a7924e4-0728-45ba-9f05-1a4e1d725062-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:53.546620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.546561 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:53.546620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.546575 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:53.546620 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.546601 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:53.548108 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.548072 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-token" pod "4a7924e4-0728-45ba-9f05-1a4e1d725062" (UID: "4a7924e4-0728-45ba-9f05-1a4e1d725062"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:53.548203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.548112 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-kube-api-access-6wbq7" pod "4a7924e4-0728-45ba-9f05-1a4e1d725062" (UID: "4a7924e4-0728-45ba-9f05-1a4e1d725062"). InnerVolumeSpecName "kube-api-access-6wbq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:53.548203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.548122 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-envoy" pod "4a7924e4-0728-45ba-9f05-1a4e1d725062" (UID: "4a7924e4-0728-45ba-9f05-1a4e1d725062"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:53:53.548203 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.548131 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-podinfo" pod "4a7924e4-0728-45ba-9f05-1a4e1d725062" (UID: "4a7924e4-0728-45ba-9f05-1a4e1d725062"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:53:53.647995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.647961 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:53.647995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.647990 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:53.647995 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.648001 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wbq7\" (UniqueName: \"kubernetes.io/projected/4a7924e4-0728-45ba-9f05-1a4e1d725062-kube-api-access-6wbq7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:53.648246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.648010 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a7924e4-0728-45ba-9f05-1a4e1d725062-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:53.873237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.873205 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf"] Oct 29 23:53:53.884669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:53.884638 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8a8b61-openshift-default-5fff8d847-xc5vf"] Oct 29 23:53:54.991739 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:54.991693 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp"] Oct 29 23:53:55.002937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.002906 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" path="/var/lib/kubelet/pods/4a7924e4-0728-45ba-9f05-1a4e1d725062/volumes" Oct 29 23:53:55.003691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.003659 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4e2ac7-j4lbp"] Oct 29 23:53:55.758710 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.758675 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g"] Oct 29 23:53:55.759194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759169 5736 cpu_manager.go:414] "containerMap: removing container" podUID="6e1f74d3-9981-4244-a824-ee910022a624" containerName="maas-api" Oct 29 23:53:55.759194 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759193 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1f74d3-9981-4244-a824-ee910022a624" containerName="maas-api" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759213 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78" containerName="payload-processing" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759219 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78" containerName="payload-processing" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759228 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fdf7b783-43f7-49aa-8594-8a328ece36e0" containerName="payload-pre-processing" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759233 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf7b783-43f7-49aa-8594-8a328ece36e0" containerName="payload-pre-processing" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759256 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759261 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759271 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a7d8dc94-b54c-4179-8011-02855b6b865a" containerName="revoke-keys" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759277 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d8dc94-b54c-4179-8011-02855b6b865a" containerName="revoke-keys" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759370 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7924e4-0728-45ba-9f05-1a4e1d725062" containerName="istio-proxy" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759389 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d8dc94-b54c-4179-8011-02855b6b865a" containerName="revoke-keys" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759398 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1f74d3-9981-4244-a824-ee910022a624" containerName="maas-api" Oct 29 23:53:55.759398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759406 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edd8f12-6d6a-4d3a-aec1-3a91e24c2b78" containerName="payload-processing" Oct 29 23:53:55.759929 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.759414 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf7b783-43f7-49aa-8594-8a328ece36e0" containerName="payload-pre-processing" Oct 29 23:53:55.764236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.764213 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:55.784887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.784858 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g"] Oct 29 23:53:55.796378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.796346 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:53:55.796522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.796414 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:53:55.869690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.869658 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:55.869829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.869691 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:55.869829 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.869783 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65q9n\" (UniqueName: \"kubernetes.io/projected/2a43194c-12e9-4b94-be3d-c538527616f8-kube-api-access-65q9n\") pod \"maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:55.970383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.970347 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65q9n\" (UniqueName: \"kubernetes.io/projected/2a43194c-12e9-4b94-be3d-c538527616f8-kube-api-access-65q9n\") pod \"maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:55.970550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.970495 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:55.970550 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.970525 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:55.971154 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.971134 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:55.972902 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.972879 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:55.993158 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:55.993117 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65q9n\" (UniqueName: \"kubernetes.io/projected/2a43194c-12e9-4b94-be3d-c538527616f8-kube-api-access-65q9n\") pod \"maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:56.077830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:56.077752 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:56.381011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:56.380924 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g"] Oct 29 23:53:56.384431 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:53:56.384393 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a43194c_12e9_4b94_be3d_c538527616f8.slice/crio-ae9b3bc47ad018903a56b98e5a0e1ce2c6701a6f90952c6d902dc30a089ee527 WatchSource:0}: Error finding container ae9b3bc47ad018903a56b98e5a0e1ce2c6701a6f90952c6d902dc30a089ee527: Status 404 returned error can't find the container with id ae9b3bc47ad018903a56b98e5a0e1ce2c6701a6f90952c6d902dc30a089ee527 Oct 29 23:53:56.549861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:56.549825 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" event={"ID":"2a43194c-12e9-4b94-be3d-c538527616f8","Type":"ContainerStarted","Data":"ae9b3bc47ad018903a56b98e5a0e1ce2c6701a6f90952c6d902dc30a089ee527"} Oct 29 23:53:56.824546 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:53:56.824507 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:53:57.003349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:57.003313 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d8dc94-b54c-4179-8011-02855b6b865a" path="/var/lib/kubelet/pods/a7d8dc94-b54c-4179-8011-02855b6b865a/volumes" Oct 29 23:53:57.557678 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:57.557644 5736 generic.go:356] "Generic (PLEG): container finished" podID="2a43194c-12e9-4b94-be3d-c538527616f8" containerID="ca734b84cce172ed0e389c039f51b67f551b1e9f4e31e2422770030886467cdf" exitCode=0 Oct 29 23:53:57.557844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:57.557734 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" event={"ID":"2a43194c-12e9-4b94-be3d-c538527616f8","Type":"ContainerDied","Data":"ca734b84cce172ed0e389c039f51b67f551b1e9f4e31e2422770030886467cdf"} Oct 29 23:53:58.703098 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.703071 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:58.803319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.803286 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-openshift-ca\") pod \"2a43194c-12e9-4b94-be3d-c538527616f8\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " Oct 29 23:53:58.803473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.803329 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a43194c-12e9-4b94-be3d-c538527616f8-kube-api-access-65q9n\" (UniqueName: \"kubernetes.io/projected/2a43194c-12e9-4b94-be3d-c538527616f8-kube-api-access-65q9n\") pod \"2a43194c-12e9-4b94-be3d-c538527616f8\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " Oct 29 23:53:58.803527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.803482 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-xks-ca\") pod \"2a43194c-12e9-4b94-be3d-c538527616f8\" (UID: \"2a43194c-12e9-4b94-be3d-c538527616f8\") " Oct 29 23:53:58.803753 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.803727 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-openshift-ca" pod "2a43194c-12e9-4b94-be3d-c538527616f8" (UID: "2a43194c-12e9-4b94-be3d-c538527616f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:53:58.803894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.803877 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:58.805523 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.805491 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a43194c-12e9-4b94-be3d-c538527616f8-kube-api-access-65q9n" pod "2a43194c-12e9-4b94-be3d-c538527616f8" (UID: "2a43194c-12e9-4b94-be3d-c538527616f8"). InnerVolumeSpecName "kube-api-access-65q9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:53:58.805671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.805527 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-xks-ca" pod "2a43194c-12e9-4b94-be3d-c538527616f8" (UID: "2a43194c-12e9-4b94-be3d-c538527616f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:53:58.904534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.904452 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a43194c-12e9-4b94-be3d-c538527616f8-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:58.904534 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:58.904484 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65q9n\" (UniqueName: \"kubernetes.io/projected/2a43194c-12e9-4b94-be3d-c538527616f8-kube-api-access-65q9n\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:53:59.571849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:59.571820 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" Oct 29 23:53:59.572027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:59.571817 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g" event={"ID":"2a43194c-12e9-4b94-be3d-c538527616f8","Type":"ContainerDied","Data":"ae9b3bc47ad018903a56b98e5a0e1ce2c6701a6f90952c6d902dc30a089ee527"} Oct 29 23:53:59.572027 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:53:59.571982 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae9b3bc47ad018903a56b98e5a0e1ce2c6701a6f90952c6d902dc30a089ee527" Oct 29 23:54:00.125566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:00.125517 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" podUID="a7efa29c-fb60-46e2-baa2-15d5c816a4e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:00.125566 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:00.125553 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" podUID="ca615e7b-9e24-4aa4-a005-bbd24c17a39a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:02.642426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.642389 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn"] Oct 29 23:54:02.642986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.642883 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2a43194c-12e9-4b94-be3d-c538527616f8" containerName="revoke-keys" Oct 29 23:54:02.642986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.642899 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a43194c-12e9-4b94-be3d-c538527616f8" containerName="revoke-keys" Oct 29 23:54:02.642986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.642972 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a43194c-12e9-4b94-be3d-c538527616f8" containerName="revoke-keys" Oct 29 23:54:02.646201 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.646183 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.660823 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.660801 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:54:02.672751 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.672725 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:54:02.676195 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.676165 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn"] Oct 29 23:54:02.749777 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.749737 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66n26\" (UniqueName: \"kubernetes.io/projected/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-kube-api-access-66n26\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.750018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.749884 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.750018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.749963 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.851352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.851315 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.851352 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.851359 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.851613 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.851428 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66n26\" (UniqueName: \"kubernetes.io/projected/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-kube-api-access-66n26\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.852003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.851974 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.853789 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.853766 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.874562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.874539 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66n26\" (UniqueName: \"kubernetes.io/projected/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-kube-api-access-66n26\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:02.959337 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:02.959308 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:03.265662 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:03.265560 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn"] Oct 29 23:54:03.269035 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:54:03.269003 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3b65e5_e7a4_4b5a_9bca_7e96993d6814.slice/crio-61b907a23d825e6e35290fd6c2a5697438323198431c676fb8efcf6e739d5ed8 WatchSource:0}: Error finding container 61b907a23d825e6e35290fd6c2a5697438323198431c676fb8efcf6e739d5ed8: Status 404 returned error can't find the container with id 61b907a23d825e6e35290fd6c2a5697438323198431c676fb8efcf6e739d5ed8 Oct 29 23:54:03.599045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:03.599007 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" event={"ID":"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814","Type":"ContainerStarted","Data":"a7b2ccbf51e812dad918d1423aad1ec071b8f8f41122d5e1a4cc6261d431da5c"} Oct 29 23:54:03.599220 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:03.599076 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" event={"ID":"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814","Type":"ContainerStarted","Data":"61b907a23d825e6e35290fd6c2a5697438323198431c676fb8efcf6e739d5ed8"} Oct 29 23:54:03.639231 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:03.639178 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" podStartSLOduration=1.639165213 podStartE2EDuration="1.639165213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:54:03.638683491 +0000 UTC m=+4240.754603106" watchObservedRunningTime="2026-10-29 23:54:03.639165213 +0000 UTC m=+4240.755084820" Oct 29 23:54:04.607722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:04.607685 5736 generic.go:356] "Generic (PLEG): container finished" podID="cb3b65e5-e7a4-4b5a-9bca-7e96993d6814" containerID="a7b2ccbf51e812dad918d1423aad1ec071b8f8f41122d5e1a4cc6261d431da5c" exitCode=0 Oct 29 23:54:04.608125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:04.607769 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" event={"ID":"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814","Type":"ContainerDied","Data":"a7b2ccbf51e812dad918d1423aad1ec071b8f8f41122d5e1a4cc6261d431da5c"} Oct 29 23:54:05.506535 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.506493 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" containerID="cri-o://a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc" gracePeriod=45 Oct 29 23:54:05.516538 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.516508 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" containerID="cri-o://0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2" gracePeriod=45 Oct 29 23:54:05.522295 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.522210 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" containerID="cri-o://70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38" gracePeriod=45 Oct 29 23:54:05.767671 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.767594 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:05.832259 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.832221 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:54:05.852093 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.852065 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:54:05.855279 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.855257 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:54:05.881062 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.881029 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-kube-api-access-66n26\" (UniqueName: \"kubernetes.io/projected/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-kube-api-access-66n26\") pod \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " Oct 29 23:54:05.881216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.881126 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-openshift-ca\") pod \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " Oct 29 23:54:05.881309 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.881224 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-xks-ca\") pod \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\" (UID: \"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814\") " Oct 29 23:54:05.881507 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.881472 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-openshift-ca" pod "cb3b65e5-e7a4-4b5a-9bca-7e96993d6814" (UID: "cb3b65e5-e7a4-4b5a-9bca-7e96993d6814"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:05.881719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.881701 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.883322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.883299 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-xks-ca" pod "cb3b65e5-e7a4-4b5a-9bca-7e96993d6814" (UID: "cb3b65e5-e7a4-4b5a-9bca-7e96993d6814"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:05.883737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.883708 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-kube-api-access-66n26" pod "cb3b65e5-e7a4-4b5a-9bca-7e96993d6814" (UID: "cb3b65e5-e7a4-4b5a-9bca-7e96993d6814"). InnerVolumeSpecName "kube-api-access-66n26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:05.982049 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982011 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-dshm\") pod \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " Oct 29 23:54:05.982255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982063 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tmp-dir\") pod \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " Oct 29 23:54:05.982255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982105 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-model-cache\") pod \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " Oct 29 23:54:05.982255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982177 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-kube-api-access-4nkgq\" (UniqueName: \"kubernetes.io/projected/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-kube-api-access-4nkgq\") pod \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " Oct 29 23:54:05.982255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982209 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6f2e1d-f61f-4da6-b374-537050d0c978-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f2e1d-f61f-4da6-b374-537050d0c978-tls-certs\") pod \"2d6f2e1d-f61f-4da6-b374-537050d0c978\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " Oct 29 23:54:05.982255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982237 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tls-certs\") pod \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982269 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-model-cache\") pod \"2d6f2e1d-f61f-4da6-b374-537050d0c978\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982297 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-home\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-home\") pod \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982329 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-model-cache\") pod \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982373 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6f2e1d-f61f-4da6-b374-537050d0c978-kube-api-access-clflq\" (UniqueName: \"kubernetes.io/projected/2d6f2e1d-f61f-4da6-b374-537050d0c978-kube-api-access-clflq\") pod \"2d6f2e1d-f61f-4da6-b374-537050d0c978\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982408 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-home\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-home\") pod \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982444 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-model-cache" pod "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" (UID: "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982451 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tmp-dir" pod "fe63a674-17d5-4a5a-ac1c-e76de31fd12d" (UID: "fe63a674-17d5-4a5a-ac1c-e76de31fd12d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982450 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-kube-api-access-nq2rf\" (UniqueName: \"kubernetes.io/projected/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-kube-api-access-nq2rf\") pod \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982533 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-dshm\") pod \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\" (UID: \"fe63a674-17d5-4a5a-ac1c-e76de31fd12d\") " Oct 29 23:54:05.982621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982602 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-home\") pod \"2d6f2e1d-f61f-4da6-b374-537050d0c978\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " Oct 29 23:54:05.983208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982722 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-model-cache" pod "2d6f2e1d-f61f-4da6-b374-537050d0c978" (UID: "2d6f2e1d-f61f-4da6-b374-537050d0c978"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.983208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982743 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tmp-dir\") pod \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " Oct 29 23:54:05.983208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982775 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-tmp-dir\") pod \"2d6f2e1d-f61f-4da6-b374-537050d0c978\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " Oct 29 23:54:05.983208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982829 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-dshm\") pod \"2d6f2e1d-f61f-4da6-b374-537050d0c978\" (UID: \"2d6f2e1d-f61f-4da6-b374-537050d0c978\") " Oct 29 23:54:05.983208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.982876 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tls-certs\" (UniqueName: \"kubernetes.io/secret/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tls-certs\") pod \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\" (UID: \"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156\") " Oct 29 23:54:05.983624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.983319 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66n26\" (UniqueName: \"kubernetes.io/projected/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-kube-api-access-66n26\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.983624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.983342 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.983624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.983359 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.983624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.983372 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.983624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.983385 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:05.984326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.984294 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-home" pod "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" (UID: "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.984669 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.984624 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-model-cache" pod "fe63a674-17d5-4a5a-ac1c-e76de31fd12d" (UID: "fe63a674-17d5-4a5a-ac1c-e76de31fd12d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.984892 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.984869 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-home" pod "fe63a674-17d5-4a5a-ac1c-e76de31fd12d" (UID: "fe63a674-17d5-4a5a-ac1c-e76de31fd12d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.985118 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.985086 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tmp-dir" pod "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" (UID: "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.985675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.985355 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-home" pod "2d6f2e1d-f61f-4da6-b374-537050d0c978" (UID: "2d6f2e1d-f61f-4da6-b374-537050d0c978"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.985675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.985431 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-tmp-dir" pod "2d6f2e1d-f61f-4da6-b374-537050d0c978" (UID: "2d6f2e1d-f61f-4da6-b374-537050d0c978"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.985675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.985605 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tls-certs" pod "fe63a674-17d5-4a5a-ac1c-e76de31fd12d" (UID: "fe63a674-17d5-4a5a-ac1c-e76de31fd12d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:05.986312 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.986274 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-kube-api-access-nq2rf" pod "fe63a674-17d5-4a5a-ac1c-e76de31fd12d" (UID: "fe63a674-17d5-4a5a-ac1c-e76de31fd12d"). InnerVolumeSpecName "kube-api-access-nq2rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:05.987202 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.987178 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-dshm" pod "fe63a674-17d5-4a5a-ac1c-e76de31fd12d" (UID: "fe63a674-17d5-4a5a-ac1c-e76de31fd12d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.988211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.988157 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-dshm" pod "2d6f2e1d-f61f-4da6-b374-537050d0c978" (UID: "2d6f2e1d-f61f-4da6-b374-537050d0c978"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.988211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.988170 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tls-certs" pod "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" (UID: "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:05.988211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.988180 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6f2e1d-f61f-4da6-b374-537050d0c978-tls-certs" pod "2d6f2e1d-f61f-4da6-b374-537050d0c978" (UID: "2d6f2e1d-f61f-4da6-b374-537050d0c978"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:05.988211 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.988177 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-dshm" pod "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" (UID: "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:05.988526 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.988476 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6f2e1d-f61f-4da6-b374-537050d0c978-kube-api-access-clflq" pod "2d6f2e1d-f61f-4da6-b374-537050d0c978" (UID: "2d6f2e1d-f61f-4da6-b374-537050d0c978"). InnerVolumeSpecName "kube-api-access-clflq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:05.989413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:05.989386 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-kube-api-access-4nkgq" pod "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" (UID: "c19011b8-e3f1-4fb5-81a8-fa98c5f0a156"). InnerVolumeSpecName "kube-api-access-4nkgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:06.085834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085736 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.085834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085775 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.085834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085791 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nkgq\" (UniqueName: \"kubernetes.io/projected/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-kube-api-access-4nkgq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.085834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085807 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6f2e1d-f61f-4da6-b374-537050d0c978-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.085834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085822 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.085834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085835 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.086306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085847 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.086306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085861 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clflq\" (UniqueName: \"kubernetes.io/projected/2d6f2e1d-f61f-4da6-b374-537050d0c978-kube-api-access-clflq\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.086306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085874 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.086306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085887 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq2rf\" (UniqueName: \"kubernetes.io/projected/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-kube-api-access-nq2rf\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.086306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085917 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe63a674-17d5-4a5a-ac1c-e76de31fd12d-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.086306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085942 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.086306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085956 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.086306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085972 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.086306 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.085983 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6f2e1d-f61f-4da6-b374-537050d0c978-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:06.622382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.622346 5736 generic.go:356] "Generic (PLEG): container finished" podID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerID="70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38" exitCode=0 Oct 29 23:54:06.622543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.622397 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" Oct 29 23:54:06.622543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.622432 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" event={"ID":"fe63a674-17d5-4a5a-ac1c-e76de31fd12d","Type":"ContainerDied","Data":"70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38"} Oct 29 23:54:06.622543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.622488 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw" event={"ID":"fe63a674-17d5-4a5a-ac1c-e76de31fd12d","Type":"ContainerDied","Data":"c6b86ebb5ad372b8a948ccfd804ff064e784be799a770b7b21efe1cc47184efd"} Oct 29 23:54:06.622543 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.622512 5736 scope.go:122] "RemoveContainer" containerID="70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38" Oct 29 23:54:06.624134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.624005 5736 generic.go:356] "Generic (PLEG): container finished" podID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerID="a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc" exitCode=0 Oct 29 23:54:06.624134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.624052 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" Oct 29 23:54:06.624134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.624085 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" event={"ID":"2d6f2e1d-f61f-4da6-b374-537050d0c978","Type":"ContainerDied","Data":"a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc"} Oct 29 23:54:06.624134 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.624123 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484" event={"ID":"2d6f2e1d-f61f-4da6-b374-537050d0c978","Type":"ContainerDied","Data":"dbd9cc48c793d70288728bf73b4426cbc3e6c21974c90e200e52bcd8a872eadb"} Oct 29 23:54:06.625788 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.625736 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" Oct 29 23:54:06.625924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.625758 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn" event={"ID":"cb3b65e5-e7a4-4b5a-9bca-7e96993d6814","Type":"ContainerDied","Data":"61b907a23d825e6e35290fd6c2a5697438323198431c676fb8efcf6e739d5ed8"} Oct 29 23:54:06.625924 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.625845 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61b907a23d825e6e35290fd6c2a5697438323198431c676fb8efcf6e739d5ed8" Oct 29 23:54:06.627408 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.627383 5736 generic.go:356] "Generic (PLEG): container finished" podID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerID="0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2" exitCode=0 Oct 29 23:54:06.627552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.627419 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" event={"ID":"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156","Type":"ContainerDied","Data":"0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2"} Oct 29 23:54:06.627552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.627446 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" event={"ID":"c19011b8-e3f1-4fb5-81a8-fa98c5f0a156","Type":"ContainerDied","Data":"9164c22b800570e88dfd796dadaf139a1bcb24cbe9ec8cb050c0f93cb3d8e0ea"} Oct 29 23:54:06.627552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.627424 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c" Oct 29 23:54:06.635832 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.635810 5736 scope.go:122] "RemoveContainer" containerID="70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38" Oct 29 23:54:06.636227 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:06.636191 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38\": container with ID starting with 70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38 not found: ID does not exist" containerID="70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38" Oct 29 23:54:06.636332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.636237 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38"} err="failed to get container status \"70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38\": rpc error: code = NotFound desc = could not find container \"70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38\": container with ID starting with 70460f0213130ee7a795a9902f26b44277d98c8fc763d6b85d06943a3e2d3e38 not found: ID does not exist" Oct 29 23:54:06.636466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.636440 5736 scope.go:122] "RemoveContainer" containerID="a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc" Oct 29 23:54:06.646839 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.646819 5736 scope.go:122] "RemoveContainer" containerID="a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc" Oct 29 23:54:06.647108 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:06.647088 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc\": container with ID starting with a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc not found: ID does not exist" containerID="a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc" Oct 29 23:54:06.647169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.647115 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc"} err="failed to get container status \"a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc\": rpc error: code = NotFound desc = could not find container \"a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc\": container with ID starting with a8c7fffb7ccc4db7ffbdfbbcd65c39c30b381ab306dc2ce3d766702d9f0652dc not found: ID does not exist" Oct 29 23:54:06.647169 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.647140 5736 scope.go:122] "RemoveContainer" containerID="0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2" Oct 29 23:54:06.655732 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.655715 5736 scope.go:122] "RemoveContainer" containerID="0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2" Oct 29 23:54:06.655970 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:06.655952 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2\": container with ID starting with 0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2 not found: ID does not exist" containerID="0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2" Oct 29 23:54:06.656025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.655977 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2"} err="failed to get container status \"0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2\": rpc error: code = NotFound desc = could not find container \"0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2\": container with ID starting with 0998b6b5862c06cb1375e6ef17b1a7973fa4eca9dd0bdec9ca961c6b7dae89b2 not found: ID does not exist" Oct 29 23:54:06.702099 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.702075 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw"] Oct 29 23:54:06.708056 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.708030 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/simulator-w5-2077b4-kserve-79546bd4db-pxmkw"] Oct 29 23:54:06.745017 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.744961 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c"] Oct 29 23:54:06.752646 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.749705 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/premium-w5-2077b4-kserve-796cdd8f59-xct2c"] Oct 29 23:54:06.781621 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.781536 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484"] Oct 29 23:54:06.784562 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:06.784532 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2077b4/distinct-w5-2077b4-kserve-7d4969db6-n5484"] Oct 29 23:54:07.005645 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.005611 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" path="/var/lib/kubelet/pods/2d6f2e1d-f61f-4da6-b374-537050d0c978/volumes" Oct 29 23:54:07.006065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.006051 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" path="/var/lib/kubelet/pods/c19011b8-e3f1-4fb5-81a8-fa98c5f0a156/volumes" Oct 29 23:54:07.006441 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.006429 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" path="/var/lib/kubelet/pods/fe63a674-17d5-4a5a-ac1c-e76de31fd12d/volumes" Oct 29 23:54:07.137142 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.137108 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:54:07.144840 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.144819 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:54:07.203322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.203293 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-kube-api-access-xzf7k\" (UniqueName: \"kubernetes.io/projected/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-kube-api-access-xzf7k\") pod \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\" (UID: \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\") " Oct 29 23:54:07.203496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.203479 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-plugins-config-volume\") pod \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\" (UID: \"ca615e7b-9e24-4aa4-a005-bbd24c17a39a\") " Oct 29 23:54:07.203868 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.203847 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-plugins-config-volume" pod "ca615e7b-9e24-4aa4-a005-bbd24c17a39a" (UID: "ca615e7b-9e24-4aa4-a005-bbd24c17a39a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:07.205426 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.205406 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-kube-api-access-xzf7k" pod "ca615e7b-9e24-4aa4-a005-bbd24c17a39a" (UID: "ca615e7b-9e24-4aa4-a005-bbd24c17a39a"). InnerVolumeSpecName "kube-api-access-xzf7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:07.304958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.304923 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-kube-api-access-2hp6w\" (UniqueName: \"kubernetes.io/projected/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-kube-api-access-2hp6w\") pod \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\" (UID: \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\") " Oct 29 23:54:07.305106 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.305011 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-plugins-config-volume\") pod \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\" (UID: \"a7efa29c-fb60-46e2-baa2-15d5c816a4e7\") " Oct 29 23:54:07.305330 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.305311 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:07.305401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.305334 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzf7k\" (UniqueName: \"kubernetes.io/projected/ca615e7b-9e24-4aa4-a005-bbd24c17a39a-kube-api-access-xzf7k\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:07.305401 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.305341 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-plugins-config-volume" pod "a7efa29c-fb60-46e2-baa2-15d5c816a4e7" (UID: "a7efa29c-fb60-46e2-baa2-15d5c816a4e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:07.307025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.307005 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-kube-api-access-2hp6w" pod "a7efa29c-fb60-46e2-baa2-15d5c816a4e7" (UID: "a7efa29c-fb60-46e2-baa2-15d5c816a4e7"). InnerVolumeSpecName "kube-api-access-2hp6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:07.406780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.406717 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hp6w\" (UniqueName: \"kubernetes.io/projected/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-kube-api-access-2hp6w\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:07.406780 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.406742 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7efa29c-fb60-46e2-baa2-15d5c816a4e7-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:07.635515 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.635480 5736 generic.go:356] "Generic (PLEG): container finished" podID="ca615e7b-9e24-4aa4-a005-bbd24c17a39a" containerID="9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49" exitCode=0 Oct 29 23:54:07.635706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.635530 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" Oct 29 23:54:07.635706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.635569 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" event={"ID":"ca615e7b-9e24-4aa4-a005-bbd24c17a39a","Type":"ContainerDied","Data":"9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49"} Oct 29 23:54:07.635706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.635648 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j" event={"ID":"ca615e7b-9e24-4aa4-a005-bbd24c17a39a","Type":"ContainerDied","Data":"1ec4775c194385b84c936651c2ae367667ed01d5b78ba1e77db1c2dfcb7f22f2"} Oct 29 23:54:07.635706 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.635680 5736 scope.go:122] "RemoveContainer" containerID="9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49" Oct 29 23:54:07.639423 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.639399 5736 generic.go:356] "Generic (PLEG): container finished" podID="a7efa29c-fb60-46e2-baa2-15d5c816a4e7" containerID="9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4" exitCode=0 Oct 29 23:54:07.639552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.639456 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" Oct 29 23:54:07.639552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.639483 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" event={"ID":"a7efa29c-fb60-46e2-baa2-15d5c816a4e7","Type":"ContainerDied","Data":"9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4"} Oct 29 23:54:07.639552 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.639534 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9" event={"ID":"a7efa29c-fb60-46e2-baa2-15d5c816a4e7","Type":"ContainerDied","Data":"fb431fcc603661c92ad946837640bd5ba640cc574615caa5bae7f287d460bc9f"} Oct 29 23:54:07.647405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.647387 5736 scope.go:122] "RemoveContainer" containerID="9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49" Oct 29 23:54:07.647691 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:07.647668 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49\": container with ID starting with 9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49 not found: ID does not exist" containerID="9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49" Oct 29 23:54:07.647783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.647698 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49"} err="failed to get container status \"9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49\": rpc error: code = NotFound desc = could not find container \"9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49\": container with ID starting with 9ef878bd90cd42eced00df527b1483f846bbfa780cabfe6952e63bb44b0b4a49 not found: ID does not exist" Oct 29 23:54:07.647783 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.647735 5736 scope.go:122] "RemoveContainer" containerID="9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4" Oct 29 23:54:07.657117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.657099 5736 scope.go:122] "RemoveContainer" containerID="9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4" Oct 29 23:54:07.657383 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:07.657361 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4\": container with ID starting with 9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4 not found: ID does not exist" containerID="9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4" Oct 29 23:54:07.657468 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.657390 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4"} err="failed to get container status \"9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4\": rpc error: code = NotFound desc = could not find container \"9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4\": container with ID starting with 9146b8b244d8fe227aa9ff415791be185eb9d9536ffc148929f8976674b1a4d4 not found: ID does not exist" Oct 29 23:54:07.686402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.686381 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9"] Oct 29 23:54:07.689466 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.689440 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4e2ac7-68796b67cf-thvn9"] Oct 29 23:54:07.722171 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.722141 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j"] Oct 29 23:54:07.739573 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:07.739535 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4e2ac7-7b96cffc4b-jv46j"] Oct 29 23:54:09.003011 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:09.002977 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7efa29c-fb60-46e2-baa2-15d5c816a4e7" path="/var/lib/kubelet/pods/a7efa29c-fb60-46e2-baa2-15d5c816a4e7/volumes" Oct 29 23:54:09.003413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:09.003372 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca615e7b-9e24-4aa4-a005-bbd24c17a39a" path="/var/lib/kubelet/pods/ca615e7b-9e24-4aa4-a005-bbd24c17a39a/volumes" Oct 29 23:54:12.884012 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.883905 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt"] Oct 29 23:54:12.884854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884824 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" Oct 29 23:54:12.884854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884848 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884868 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a7efa29c-fb60-46e2-baa2-15d5c816a4e7" containerName="payload-pre-processing" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884874 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7efa29c-fb60-46e2-baa2-15d5c816a4e7" containerName="payload-pre-processing" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884884 5736 cpu_manager.go:414] "containerMap: removing container" podUID="cb3b65e5-e7a4-4b5a-9bca-7e96993d6814" containerName="revoke-keys" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884890 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3b65e5-e7a4-4b5a-9bca-7e96993d6814" containerName="revoke-keys" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884910 5736 cpu_manager.go:414] "containerMap: removing container" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884914 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884925 5736 cpu_manager.go:414] "containerMap: removing container" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884930 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884940 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ca615e7b-9e24-4aa4-a005-bbd24c17a39a" containerName="payload-processing" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884946 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca615e7b-9e24-4aa4-a005-bbd24c17a39a" containerName="payload-processing" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.884996 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7efa29c-fb60-46e2-baa2-15d5c816a4e7" containerName="payload-pre-processing" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.885005 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3b65e5-e7a4-4b5a-9bca-7e96993d6814" containerName="revoke-keys" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.885014 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6f2e1d-f61f-4da6-b374-537050d0c978" containerName="main" Oct 29 23:54:12.885015 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.885022 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe63a674-17d5-4a5a-ac1c-e76de31fd12d" containerName="main" Oct 29 23:54:12.885654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.885029 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca615e7b-9e24-4aa4-a005-bbd24c17a39a" containerName="payload-processing" Oct 29 23:54:12.885654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.885035 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19011b8-e3f1-4fb5-81a8-fa98c5f0a156" containerName="main" Oct 29 23:54:12.889624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.889606 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:12.909602 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.909554 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt"] Oct 29 23:54:12.916052 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.916031 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:54:12.916153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.916047 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:54:12.966983 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.966952 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptw4f\" (UniqueName: \"kubernetes.io/projected/c264a13b-b2a8-47eb-ac9f-91fd56328700-kube-api-access-ptw4f\") pod \"maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:12.967086 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.966992 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:12.967144 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:12.967089 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:13.068275 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.068242 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptw4f\" (UniqueName: \"kubernetes.io/projected/c264a13b-b2a8-47eb-ac9f-91fd56328700-kube-api-access-ptw4f\") pod \"maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:13.068447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.068284 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:13.068447 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.068325 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:13.068937 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.068918 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:13.070705 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.070682 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:13.091359 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.091331 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptw4f\" (UniqueName: \"kubernetes.io/projected/c264a13b-b2a8-47eb-ac9f-91fd56328700-kube-api-access-ptw4f\") pod \"maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:13.202891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.202821 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:13.421699 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.421644 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm"] Oct 29 23:54:13.505472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.505434 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt"] Oct 29 23:54:13.511784 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:54:13.511749 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc264a13b_b2a8_47eb_ac9f_91fd56328700.slice/crio-1e066e2865e92df548fa51242c9cc8bf958f26b9e172aebdc68200cacb7b7615 WatchSource:0}: Error finding container 1e066e2865e92df548fa51242c9cc8bf958f26b9e172aebdc68200cacb7b7615: Status 404 returned error can't find the container with id 1e066e2865e92df548fa51242c9cc8bf958f26b9e172aebdc68200cacb7b7615 Oct 29 23:54:13.687112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:13.687079 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" event={"ID":"c264a13b-b2a8-47eb-ac9f-91fd56328700","Type":"ContainerStarted","Data":"1e066e2865e92df548fa51242c9cc8bf958f26b9e172aebdc68200cacb7b7615"} Oct 29 23:54:14.034522 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:14.034482 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss"] Oct 29 23:54:14.036274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:14.036244 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn"] Oct 29 23:54:14.045498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:14.045474 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab28727-q82ss"] Oct 29 23:54:14.047245 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:14.047221 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85f3e908-qv5kn"] Oct 29 23:54:14.696218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:14.696126 5736 generic.go:356] "Generic (PLEG): container finished" podID="c264a13b-b2a8-47eb-ac9f-91fd56328700" containerID="a4eecc32d2e9793a560ae69e0e01ca70b2ced6bdb689f61cc3ea2a5d0ff3827b" exitCode=0 Oct 29 23:54:14.696218 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:14.696164 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" event={"ID":"c264a13b-b2a8-47eb-ac9f-91fd56328700","Type":"ContainerDied","Data":"a4eecc32d2e9793a560ae69e0e01ca70b2ced6bdb689f61cc3ea2a5d0ff3827b"} Oct 29 23:54:15.009332 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.009297 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6bca83-635f-49f1-8485-0033968f6318" path="/var/lib/kubelet/pods/0a6bca83-635f-49f1-8485-0033968f6318/volumes" Oct 29 23:54:15.009748 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.009733 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1" path="/var/lib/kubelet/pods/15e8aa16-e4ee-4c0c-ae66-f547b1fcc2d1/volumes" Oct 29 23:54:15.845274 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.845247 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:15.996895 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.996858 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-xks-ca\") pod \"c264a13b-b2a8-47eb-ac9f-91fd56328700\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " Oct 29 23:54:15.997079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.996919 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-openshift-ca\") pod \"c264a13b-b2a8-47eb-ac9f-91fd56328700\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " Oct 29 23:54:15.997123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.997112 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c264a13b-b2a8-47eb-ac9f-91fd56328700-kube-api-access-ptw4f\" (UniqueName: \"kubernetes.io/projected/c264a13b-b2a8-47eb-ac9f-91fd56328700-kube-api-access-ptw4f\") pod \"c264a13b-b2a8-47eb-ac9f-91fd56328700\" (UID: \"c264a13b-b2a8-47eb-ac9f-91fd56328700\") " Oct 29 23:54:15.997302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.997277 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-openshift-ca" pod "c264a13b-b2a8-47eb-ac9f-91fd56328700" (UID: "c264a13b-b2a8-47eb-ac9f-91fd56328700"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:15.997477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.997463 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:15.998999 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.998979 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-xks-ca" pod "c264a13b-b2a8-47eb-ac9f-91fd56328700" (UID: "c264a13b-b2a8-47eb-ac9f-91fd56328700"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:15.999243 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:15.999210 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c264a13b-b2a8-47eb-ac9f-91fd56328700-kube-api-access-ptw4f" pod "c264a13b-b2a8-47eb-ac9f-91fd56328700" (UID: "c264a13b-b2a8-47eb-ac9f-91fd56328700"). InnerVolumeSpecName "kube-api-access-ptw4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:16.098497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.098464 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c264a13b-b2a8-47eb-ac9f-91fd56328700-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:16.098497 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.098494 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptw4f\" (UniqueName: \"kubernetes.io/projected/c264a13b-b2a8-47eb-ac9f-91fd56328700-kube-api-access-ptw4f\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:16.712464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.712432 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" Oct 29 23:54:16.712464 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.712453 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt" event={"ID":"c264a13b-b2a8-47eb-ac9f-91fd56328700","Type":"ContainerDied","Data":"1e066e2865e92df548fa51242c9cc8bf958f26b9e172aebdc68200cacb7b7615"} Oct 29 23:54:16.712704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.712497 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e066e2865e92df548fa51242c9cc8bf958f26b9e172aebdc68200cacb7b7615" Oct 29 23:54:16.758884 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.758837 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd"] Oct 29 23:54:16.759435 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.759414 5736 cpu_manager.go:414] "containerMap: removing container" podUID="c264a13b-b2a8-47eb-ac9f-91fd56328700" containerName="revoke-keys" Oct 29 23:54:16.759505 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.759437 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c264a13b-b2a8-47eb-ac9f-91fd56328700" containerName="revoke-keys" Oct 29 23:54:16.759545 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.759513 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="c264a13b-b2a8-47eb-ac9f-91fd56328700" containerName="revoke-keys" Oct 29 23:54:16.762946 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.762919 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:16.783387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.783364 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:54:16.783509 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.783432 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:54:16.786089 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.786065 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd"] Oct 29 23:54:16.906119 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.906082 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e17667bf-kpwqd\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:16.906495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.906158 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e17667bf-kpwqd\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:16.906495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:16.906217 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkcfv\" (UniqueName: \"kubernetes.io/projected/4317d6bc-77d6-4fae-b428-604f09fc2e58-kube-api-access-gkcfv\") pod \"maas-api-revoke-sub-e17667bf-kpwqd\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:17.007663 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.007558 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkcfv\" (UniqueName: \"kubernetes.io/projected/4317d6bc-77d6-4fae-b428-604f09fc2e58-kube-api-access-gkcfv\") pod \"maas-api-revoke-sub-e17667bf-kpwqd\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:17.007850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.007680 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e17667bf-kpwqd\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:17.007850 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.007747 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e17667bf-kpwqd\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:17.008366 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.008344 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e17667bf-kpwqd\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:17.010010 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.009987 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e17667bf-kpwqd\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:17.031003 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.030974 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkcfv\" (UniqueName: \"kubernetes.io/projected/4317d6bc-77d6-4fae-b428-604f09fc2e58-kube-api-access-gkcfv\") pod \"maas-api-revoke-sub-e17667bf-kpwqd\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:17.076993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.076970 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:17.377959 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.377875 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd"] Oct 29 23:54:17.381491 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:54:17.381463 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4317d6bc_77d6_4fae_b428_604f09fc2e58.slice/crio-08ad3179e007e4d4bf1a089e69093876bcd4582996e0d8987e45850c79ea284b WatchSource:0}: Error finding container 08ad3179e007e4d4bf1a089e69093876bcd4582996e0d8987e45850c79ea284b: Status 404 returned error can't find the container with id 08ad3179e007e4d4bf1a089e69093876bcd4582996e0d8987e45850c79ea284b Oct 29 23:54:17.494253 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.494222 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv"] Oct 29 23:54:17.522556 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.522526 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn"] Oct 29 23:54:17.720729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.720697 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" event={"ID":"4317d6bc-77d6-4fae-b428-604f09fc2e58","Type":"ContainerStarted","Data":"f559f650bce4ba8c21785970aaad6c76cda1deac6989a1adc3735fa586b54be8"} Oct 29 23:54:17.720882 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.720748 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" event={"ID":"4317d6bc-77d6-4fae-b428-604f09fc2e58","Type":"ContainerStarted","Data":"08ad3179e007e4d4bf1a089e69093876bcd4582996e0d8987e45850c79ea284b"} Oct 29 23:54:17.761357 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:17.761309 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" podStartSLOduration=1.761293768 podStartE2EDuration="1.761293768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:54:17.759715324 +0000 UTC m=+4254.875634929" watchObservedRunningTime="2026-10-29 23:54:17.761293768 +0000 UTC m=+4254.877213373" Oct 29 23:54:18.423009 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:18.422920 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" podUID="ab7cc703-61ff-4a81-b215-a5619f21af8a" containerName="payload-pre-processing" containerID="cri-o://a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510" gracePeriod=25 Oct 29 23:54:18.729038 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:18.729000 5736 generic.go:356] "Generic (PLEG): container finished" podID="4317d6bc-77d6-4fae-b428-604f09fc2e58" containerID="f559f650bce4ba8c21785970aaad6c76cda1deac6989a1adc3735fa586b54be8" exitCode=0 Oct 29 23:54:18.729215 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:18.729084 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" event={"ID":"4317d6bc-77d6-4fae-b428-604f09fc2e58","Type":"ContainerDied","Data":"f559f650bce4ba8c21785970aaad6c76cda1deac6989a1adc3735fa586b54be8"} Oct 29 23:54:19.861414 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:19.861387 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:20.040448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.040363 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-openshift-ca\") pod \"4317d6bc-77d6-4fae-b428-604f09fc2e58\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " Oct 29 23:54:20.040652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.040453 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4317d6bc-77d6-4fae-b428-604f09fc2e58-kube-api-access-gkcfv\" (UniqueName: \"kubernetes.io/projected/4317d6bc-77d6-4fae-b428-604f09fc2e58-kube-api-access-gkcfv\") pod \"4317d6bc-77d6-4fae-b428-604f09fc2e58\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " Oct 29 23:54:20.040652 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.040481 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-xks-ca\") pod \"4317d6bc-77d6-4fae-b428-604f09fc2e58\" (UID: \"4317d6bc-77d6-4fae-b428-604f09fc2e58\") " Oct 29 23:54:20.040799 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.040759 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-openshift-ca" pod "4317d6bc-77d6-4fae-b428-604f09fc2e58" (UID: "4317d6bc-77d6-4fae-b428-604f09fc2e58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:20.042615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.042572 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4317d6bc-77d6-4fae-b428-604f09fc2e58-kube-api-access-gkcfv" pod "4317d6bc-77d6-4fae-b428-604f09fc2e58" (UID: "4317d6bc-77d6-4fae-b428-604f09fc2e58"). InnerVolumeSpecName "kube-api-access-gkcfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:20.042744 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.042591 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-xks-ca" pod "4317d6bc-77d6-4fae-b428-604f09fc2e58" (UID: "4317d6bc-77d6-4fae-b428-604f09fc2e58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:20.142452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.142418 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:20.142452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.142446 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkcfv\" (UniqueName: \"kubernetes.io/projected/4317d6bc-77d6-4fae-b428-604f09fc2e58-kube-api-access-gkcfv\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:20.142452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.142457 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4317d6bc-77d6-4fae-b428-604f09fc2e58-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:20.743838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.743805 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" Oct 29 23:54:20.744002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.743837 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17667bf-kpwqd" event={"ID":"4317d6bc-77d6-4fae-b428-604f09fc2e58","Type":"ContainerDied","Data":"08ad3179e007e4d4bf1a089e69093876bcd4582996e0d8987e45850c79ea284b"} Oct 29 23:54:20.744002 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:20.743880 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08ad3179e007e4d4bf1a089e69093876bcd4582996e0d8987e45850c79ea284b" Oct 29 23:54:21.099863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:21.099785 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn"] Oct 29 23:54:21.105129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:21.105097 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2126fb-llmdn"] Oct 29 23:54:23.003729 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:23.003695 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3b65e5-e7a4-4b5a-9bca-7e96993d6814" path="/var/lib/kubelet/pods/cb3b65e5-e7a4-4b5a-9bca-7e96993d6814/volumes" Oct 29 23:54:24.980593 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:24.980540 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" podUID="ab7cc703-61ff-4a81-b215-a5619f21af8a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:26.381008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.380970 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v"] Oct 29 23:54:26.381542 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.381520 5736 cpu_manager.go:414] "containerMap: removing container" podUID="4317d6bc-77d6-4fae-b428-604f09fc2e58" containerName="revoke-keys" Oct 29 23:54:26.381674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.381542 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4317d6bc-77d6-4fae-b428-604f09fc2e58" containerName="revoke-keys" Oct 29 23:54:26.381674 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.381646 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="4317d6bc-77d6-4fae-b428-604f09fc2e58" containerName="revoke-keys" Oct 29 23:54:26.386080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.386060 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.404191 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.404162 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:54:26.409842 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.409817 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2816861c-wdg2v\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.409956 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.409869 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2816861c-wdg2v\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.410035 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.410007 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpf7\" (UniqueName: \"kubernetes.io/projected/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-kube-api-access-gmpf7\") pod \"maas-api-revoke-sub-2816861c-wdg2v\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.416450 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.416424 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\"" Oct 29 23:54:26.418691 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.418665 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v"] Oct 29 23:54:26.511491 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.511449 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2816861c-wdg2v\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.511722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.511529 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2816861c-wdg2v\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.511722 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.511561 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpf7\" (UniqueName: \"kubernetes.io/projected/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-kube-api-access-gmpf7\") pod \"maas-api-revoke-sub-2816861c-wdg2v\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.512496 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.512469 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2816861c-wdg2v\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.514794 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.514771 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2816861c-wdg2v\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.536358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.536305 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpf7\" (UniqueName: \"kubernetes.io/projected/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-kube-api-access-gmpf7\") pod \"maas-api-revoke-sub-2816861c-wdg2v\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.699928 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:26.699858 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:26.824718 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:26.824677 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:54:27.009898 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:27.009510 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v"] Oct 29 23:54:27.014678 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:54:27.014640 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8855fdf9_1a85_4364_a2d6_10a9278b9bc4.slice/crio-7a9a19aa9587d3eb42e81c3f4385f0b662c80e43cc132c66d22341bf3bcfdc56 WatchSource:0}: Error finding container 7a9a19aa9587d3eb42e81c3f4385f0b662c80e43cc132c66d22341bf3bcfdc56: Status 404 returned error can't find the container with id 7a9a19aa9587d3eb42e81c3f4385f0b662c80e43cc132c66d22341bf3bcfdc56 Oct 29 23:54:27.799651 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:27.799564 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" event={"ID":"8855fdf9-1a85-4364-a2d6-10a9278b9bc4","Type":"ContainerStarted","Data":"9fcb7a6730532c410ac2855ec131483e9131b065fb07107689ab6400b66f4214"} Oct 29 23:54:27.800136 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:27.799692 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" event={"ID":"8855fdf9-1a85-4364-a2d6-10a9278b9bc4","Type":"ContainerStarted","Data":"7a9a19aa9587d3eb42e81c3f4385f0b662c80e43cc132c66d22341bf3bcfdc56"} Oct 29 23:54:27.844807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:27.844760 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" podStartSLOduration=1.84474839 podStartE2EDuration="1.84474839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:54:27.843953058 +0000 UTC m=+4264.959872663" watchObservedRunningTime="2026-10-29 23:54:27.84474839 +0000 UTC m=+4264.960667993" Oct 29 23:54:28.809433 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:28.809396 5736 generic.go:356] "Generic (PLEG): container finished" podID="8855fdf9-1a85-4364-a2d6-10a9278b9bc4" containerID="9fcb7a6730532c410ac2855ec131483e9131b065fb07107689ab6400b66f4214" exitCode=0 Oct 29 23:54:28.809855 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:28.809481 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" event={"ID":"8855fdf9-1a85-4364-a2d6-10a9278b9bc4","Type":"ContainerDied","Data":"9fcb7a6730532c410ac2855ec131483e9131b065fb07107689ab6400b66f4214"} Oct 29 23:54:29.149704 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:29.149613 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g"] Oct 29 23:54:29.152871 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:29.152834 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2077b4-tqq4g"] Oct 29 23:54:29.959322 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:29.959297 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:30.060569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.060528 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-xks-ca\") pod \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " Oct 29 23:54:30.060807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.060625 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-kube-api-access-gmpf7\" (UniqueName: \"kubernetes.io/projected/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-kube-api-access-gmpf7\") pod \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " Oct 29 23:54:30.060807 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.060757 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-openshift-ca\") pod \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\" (UID: \"8855fdf9-1a85-4364-a2d6-10a9278b9bc4\") " Oct 29 23:54:30.061252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.061219 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-openshift-ca" pod "8855fdf9-1a85-4364-a2d6-10a9278b9bc4" (UID: "8855fdf9-1a85-4364-a2d6-10a9278b9bc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:30.061656 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.061626 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-openshift-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:30.062996 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.062973 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-xks-ca" pod "8855fdf9-1a85-4364-a2d6-10a9278b9bc4" (UID: "8855fdf9-1a85-4364-a2d6-10a9278b9bc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:30.063176 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.063149 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-kube-api-access-gmpf7" pod "8855fdf9-1a85-4364-a2d6-10a9278b9bc4" (UID: "8855fdf9-1a85-4364-a2d6-10a9278b9bc4"). InnerVolumeSpecName "kube-api-access-gmpf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:30.163190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.163155 5736 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-maas-api-xks-ca\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:30.163190 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.163185 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmpf7\" (UniqueName: \"kubernetes.io/projected/8855fdf9-1a85-4364-a2d6-10a9278b9bc4-kube-api-access-gmpf7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:30.824040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.824009 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" Oct 29 23:54:30.824040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.824022 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2816861c-wdg2v" event={"ID":"8855fdf9-1a85-4364-a2d6-10a9278b9bc4","Type":"ContainerDied","Data":"7a9a19aa9587d3eb42e81c3f4385f0b662c80e43cc132c66d22341bf3bcfdc56"} Oct 29 23:54:30.824258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.824065 5736 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a9a19aa9587d3eb42e81c3f4385f0b662c80e43cc132c66d22341bf3bcfdc56" Oct 29 23:54:30.990830 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.990793 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj"] Oct 29 23:54:30.991320 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:30.991047 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerName="istio-proxy" containerID="cri-o://e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289" gracePeriod=30 Oct 29 23:54:31.005131 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:31.005091 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a43194c-12e9-4b94-be3d-c538527616f8" path="/var/lib/kubelet/pods/2a43194c-12e9-4b94-be3d-c538527616f8/volumes" Oct 29 23:54:31.818715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:31.818667 5736 scope.go:122] "RemoveContainer" containerID="d24cad7f1021d31366007feac07d318bc546abe7ae679e9920a613e8b1e629ac" Oct 29 23:54:31.834063 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:31.834026 5736 scope.go:122] "RemoveContainer" containerID="3d435b3a393decfd05b555e892787264c91beb31e37a364307e0ec218d75a1c7" Oct 29 23:54:31.843559 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:31.843538 5736 scope.go:122] "RemoveContainer" containerID="b445121a50da7ef0504f294fffbbd666c913470a81e0898222fb93f9d27ed3fe" Oct 29 23:54:31.852605 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:31.852556 5736 scope.go:122] "RemoveContainer" containerID="1c10bfb1b7ec2a16ff2e45a54d644952acfc7da7eeaf6528f0d1db425242027f" Oct 29 23:54:31.861537 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:31.861515 5736 scope.go:122] "RemoveContainer" containerID="94a4654718fde719613cbb75e5a6d93184eeba271a184ae6ef70e363d5b6f377" Oct 29 23:54:32.514541 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.514499 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" containerID="cri-o://ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c" gracePeriod=45 Oct 29 23:54:32.543825 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.543790 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" containerID="cri-o://ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6" gracePeriod=45 Oct 29 23:54:32.788923 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.788892 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:54:32.815126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.815103 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 23:54:32.845921 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.845888 5736 generic.go:356] "Generic (PLEG): container finished" podID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerID="ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c" exitCode=0 Oct 29 23:54:32.846082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.845939 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" Oct 29 23:54:32.846082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.845966 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" event={"ID":"1ea31536-fd5e-489b-8092-8d9a8cb816cd","Type":"ContainerDied","Data":"ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c"} Oct 29 23:54:32.846082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.846008 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv" event={"ID":"1ea31536-fd5e-489b-8092-8d9a8cb816cd","Type":"ContainerDied","Data":"293a6d518114fc1167bdd902379c25215cb44c2993d98a46e7580ecc2cf1f33f"} Oct 29 23:54:32.846082 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.846041 5736 scope.go:122] "RemoveContainer" containerID="ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c" Oct 29 23:54:32.847761 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.847735 5736 generic.go:356] "Generic (PLEG): container finished" podID="902df38e-ddc0-457e-8877-2702115f3de8" containerID="ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6" exitCode=0 Oct 29 23:54:32.847886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.847775 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" event={"ID":"902df38e-ddc0-457e-8877-2702115f3de8","Type":"ContainerDied","Data":"ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6"} Oct 29 23:54:32.847886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.847780 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" Oct 29 23:54:32.847886 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.847796 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn" event={"ID":"902df38e-ddc0-457e-8877-2702115f3de8","Type":"ContainerDied","Data":"c2d96713c0a4231241dee6530086329f82e8edd769c1f25e88200116cfac1003"} Oct 29 23:54:32.857350 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.857324 5736 scope.go:122] "RemoveContainer" containerID="ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c" Oct 29 23:54:32.858007 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:32.857815 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c\": container with ID starting with ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c not found: ID does not exist" containerID="ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c" Oct 29 23:54:32.858007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.857885 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c"} err="failed to get container status \"ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c\": rpc error: code = NotFound desc = could not find container \"ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c\": container with ID starting with ad4c9d6b07650efbb49cbf23eef8e93bf8757c4d5aca594dadf7bd368e17c10c not found: ID does not exist" Oct 29 23:54:32.858007 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.857910 5736 scope.go:122] "RemoveContainer" containerID="ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6" Oct 29 23:54:32.870834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.870804 5736 scope.go:122] "RemoveContainer" containerID="ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6" Oct 29 23:54:32.871446 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:32.871410 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6\": container with ID starting with ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6 not found: ID does not exist" containerID="ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6" Oct 29 23:54:32.871957 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.871453 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6"} err="failed to get container status \"ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6\": rpc error: code = NotFound desc = could not find container \"ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6\": container with ID starting with ffd64795baeb5b02eb1be8403fdd680cc5b5e81f32bbff659f09e66f6618ebb6 not found: ID does not exist" Oct 29 23:54:32.892297 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892268 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-tmp-dir\") pod \"902df38e-ddc0-457e-8877-2702115f3de8\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " Oct 29 23:54:32.892439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892313 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-home\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-home\") pod \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " Oct 29 23:54:32.892439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892348 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902df38e-ddc0-457e-8877-2702115f3de8-kube-api-access-xzzk7\" (UniqueName: \"kubernetes.io/projected/902df38e-ddc0-457e-8877-2702115f3de8-kube-api-access-xzzk7\") pod \"902df38e-ddc0-457e-8877-2702115f3de8\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " Oct 29 23:54:32.892439 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892423 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tls-certs\") pod \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " Oct 29 23:54:32.892675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892503 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-dshm\") pod \"902df38e-ddc0-457e-8877-2702115f3de8\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " Oct 29 23:54:32.892675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892556 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-home\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-home\") pod \"902df38e-ddc0-457e-8877-2702115f3de8\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " Oct 29 23:54:32.892675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892598 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902df38e-ddc0-457e-8877-2702115f3de8-tls-certs\" (UniqueName: \"kubernetes.io/secret/902df38e-ddc0-457e-8877-2702115f3de8-tls-certs\") pod \"902df38e-ddc0-457e-8877-2702115f3de8\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " Oct 29 23:54:32.892675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892566 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-tmp-dir" pod "902df38e-ddc0-457e-8877-2702115f3de8" (UID: "902df38e-ddc0-457e-8877-2702115f3de8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:32.892675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892620 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-home" pod "1ea31536-fd5e-489b-8092-8d9a8cb816cd" (UID: "1ea31536-fd5e-489b-8092-8d9a8cb816cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:32.892675 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892639 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-model-cache\") pod \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " Oct 29 23:54:32.893023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892825 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-home" pod "902df38e-ddc0-457e-8877-2702115f3de8" (UID: "902df38e-ddc0-457e-8877-2702115f3de8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:32.893023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892832 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-model-cache" pod "1ea31536-fd5e-489b-8092-8d9a8cb816cd" (UID: "1ea31536-fd5e-489b-8092-8d9a8cb816cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:32.893023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892861 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea31536-fd5e-489b-8092-8d9a8cb816cd-kube-api-access-bkt2n\" (UniqueName: \"kubernetes.io/projected/1ea31536-fd5e-489b-8092-8d9a8cb816cd-kube-api-access-bkt2n\") pod \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " Oct 29 23:54:32.893023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892892 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-dshm\") pod \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " Oct 29 23:54:32.893023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892912 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tmp-dir\") pod \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\" (UID: \"1ea31536-fd5e-489b-8092-8d9a8cb816cd\") " Oct 29 23:54:32.893023 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.892968 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-model-cache\") pod \"902df38e-ddc0-457e-8877-2702115f3de8\" (UID: \"902df38e-ddc0-457e-8877-2702115f3de8\") " Oct 29 23:54:32.893402 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.893285 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tmp-dir" pod "1ea31536-fd5e-489b-8092-8d9a8cb816cd" (UID: "1ea31536-fd5e-489b-8092-8d9a8cb816cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:32.893472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.893416 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.893472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.893438 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.893472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.893453 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.893472 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.893466 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.893719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.893479 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.893719 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.893540 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-model-cache" pod "902df38e-ddc0-457e-8877-2702115f3de8" (UID: "902df38e-ddc0-457e-8877-2702115f3de8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:32.895375 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.895347 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902df38e-ddc0-457e-8877-2702115f3de8-kube-api-access-xzzk7" pod "902df38e-ddc0-457e-8877-2702115f3de8" (UID: "902df38e-ddc0-457e-8877-2702115f3de8"). InnerVolumeSpecName "kube-api-access-xzzk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:32.895502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.895434 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-dshm" pod "902df38e-ddc0-457e-8877-2702115f3de8" (UID: "902df38e-ddc0-457e-8877-2702115f3de8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:32.895686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.895662 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea31536-fd5e-489b-8092-8d9a8cb816cd-kube-api-access-bkt2n" pod "1ea31536-fd5e-489b-8092-8d9a8cb816cd" (UID: "1ea31536-fd5e-489b-8092-8d9a8cb816cd"). InnerVolumeSpecName "kube-api-access-bkt2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:32.896025 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.896007 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tls-certs" pod "1ea31536-fd5e-489b-8092-8d9a8cb816cd" (UID: "1ea31536-fd5e-489b-8092-8d9a8cb816cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:32.896404 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.896386 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-dshm" pod "1ea31536-fd5e-489b-8092-8d9a8cb816cd" (UID: "1ea31536-fd5e-489b-8092-8d9a8cb816cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:32.896499 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.896472 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902df38e-ddc0-457e-8877-2702115f3de8-tls-certs" pod "902df38e-ddc0-457e-8877-2702115f3de8" (UID: "902df38e-ddc0-457e-8877-2702115f3de8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:54:32.994680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.994644 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902df38e-ddc0-457e-8877-2702115f3de8-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.994680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.994676 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkt2n\" (UniqueName: \"kubernetes.io/projected/1ea31536-fd5e-489b-8092-8d9a8cb816cd-kube-api-access-bkt2n\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.994680 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.994686 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea31536-fd5e-489b-8092-8d9a8cb816cd-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.994933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.994696 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.994933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.994706 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzzk7\" (UniqueName: \"kubernetes.io/projected/902df38e-ddc0-457e-8877-2702115f3de8-kube-api-access-xzzk7\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.994933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.994713 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea31536-fd5e-489b-8092-8d9a8cb816cd-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:32.994933 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:32.994721 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902df38e-ddc0-457e-8877-2702115f3de8-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:33.191985 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:33.191950 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv"] Oct 29 23:54:33.204251 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:33.204218 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-2-w3-32c192-kserve-d69f57d6c-ck9fv"] Oct 29 23:54:33.238024 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:33.237993 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn"] Oct 29 23:54:33.242290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:33.242265 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/simulator-w3-32c192-kserve-85c885c48d-gxkrn"] Oct 29 23:54:34.980771 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:34.980728 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" podUID="ab7cc703-61ff-4a81-b215-a5619f21af8a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:35.002331 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:35.002297 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" path="/var/lib/kubelet/pods/1ea31536-fd5e-489b-8092-8d9a8cb816cd/volumes" Oct 29 23:54:35.002778 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:35.002762 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902df38e-ddc0-457e-8877-2702115f3de8" path="/var/lib/kubelet/pods/902df38e-ddc0-457e-8877-2702115f3de8/volumes" Oct 29 23:54:36.238634 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.238611 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:54:36.325030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.324942 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09ce4d3-71b2-458f-ad73-542a86c10268-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09ce4d3-71b2-458f-ad73-542a86c10268-istiod-ca-cert\") pod \"f09ce4d3-71b2-458f-ad73-542a86c10268\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " Oct 29 23:54:36.325030 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325011 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-token\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-token\") pod \"f09ce4d3-71b2-458f-ad73-542a86c10268\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " Oct 29 23:54:36.325290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325192 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-envoy\") pod \"f09ce4d3-71b2-458f-ad73-542a86c10268\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " Oct 29 23:54:36.325290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325237 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-certs\") pod \"f09ce4d3-71b2-458f-ad73-542a86c10268\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " Oct 29 23:54:36.325290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325277 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-data\") pod \"f09ce4d3-71b2-458f-ad73-542a86c10268\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " Oct 29 23:54:36.325487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325340 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-kube-api-access-dddsb\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-kube-api-access-dddsb\") pod \"f09ce4d3-71b2-458f-ad73-542a86c10268\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " Oct 29 23:54:36.325487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325356 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09ce4d3-71b2-458f-ad73-542a86c10268-istiod-ca-cert" pod "f09ce4d3-71b2-458f-ad73-542a86c10268" (UID: "f09ce4d3-71b2-458f-ad73-542a86c10268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:36.325487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325370 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-socket\") pod \"f09ce4d3-71b2-458f-ad73-542a86c10268\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " Oct 29 23:54:36.325487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325413 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-credential-socket\") pod \"f09ce4d3-71b2-458f-ad73-542a86c10268\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " Oct 29 23:54:36.325487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325457 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-podinfo\") pod \"f09ce4d3-71b2-458f-ad73-542a86c10268\" (UID: \"f09ce4d3-71b2-458f-ad73-542a86c10268\") " Oct 29 23:54:36.325854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325535 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-certs" pod "f09ce4d3-71b2-458f-ad73-542a86c10268" (UID: "f09ce4d3-71b2-458f-ad73-542a86c10268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:36.325947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325922 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09ce4d3-71b2-458f-ad73-542a86c10268-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:36.326045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325941 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-credential-socket" pod "f09ce4d3-71b2-458f-ad73-542a86c10268" (UID: "f09ce4d3-71b2-458f-ad73-542a86c10268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:36.326045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.325950 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:36.326045 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.326030 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-socket" pod "f09ce4d3-71b2-458f-ad73-542a86c10268" (UID: "f09ce4d3-71b2-458f-ad73-542a86c10268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:36.326230 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.326085 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-data" pod "f09ce4d3-71b2-458f-ad73-542a86c10268" (UID: "f09ce4d3-71b2-458f-ad73-542a86c10268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:36.327403 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.327371 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-token" pod "f09ce4d3-71b2-458f-ad73-542a86c10268" (UID: "f09ce4d3-71b2-458f-ad73-542a86c10268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:36.327657 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.327640 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-podinfo" pod "f09ce4d3-71b2-458f-ad73-542a86c10268" (UID: "f09ce4d3-71b2-458f-ad73-542a86c10268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:54:36.327784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.327768 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-kube-api-access-dddsb" pod "f09ce4d3-71b2-458f-ad73-542a86c10268" (UID: "f09ce4d3-71b2-458f-ad73-542a86c10268"). InnerVolumeSpecName "kube-api-access-dddsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:36.328442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.328422 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-envoy" pod "f09ce4d3-71b2-458f-ad73-542a86c10268" (UID: "f09ce4d3-71b2-458f-ad73-542a86c10268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:36.426487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.426455 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:36.426487 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.426487 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:36.426720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.426500 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:36.426720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.426513 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dddsb\" (UniqueName: \"kubernetes.io/projected/f09ce4d3-71b2-458f-ad73-542a86c10268-kube-api-access-dddsb\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:36.426720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.426527 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:36.426720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.426540 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ce4d3-71b2-458f-ad73-542a86c10268-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:36.426720 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.426554 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09ce4d3-71b2-458f-ad73-542a86c10268-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:36.876126 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.876090 5736 generic.go:356] "Generic (PLEG): container finished" podID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerID="e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289" exitCode=0 Oct 29 23:54:36.876333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.876136 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" Oct 29 23:54:36.876333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.876168 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" event={"ID":"f09ce4d3-71b2-458f-ad73-542a86c10268","Type":"ContainerDied","Data":"e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289"} Oct 29 23:54:36.876333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.876233 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj" event={"ID":"f09ce4d3-71b2-458f-ad73-542a86c10268","Type":"ContainerDied","Data":"15971c90222b6008d63bdbee513903b597d093f0f005f602a0a97f7e011b702b"} Oct 29 23:54:36.876333 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.876264 5736 scope.go:122] "RemoveContainer" containerID="e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289" Oct 29 23:54:36.893473 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.893446 5736 scope.go:122] "RemoveContainer" containerID="e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289" Oct 29 23:54:36.893932 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:36.893857 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289\": container with ID starting with e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289 not found: ID does not exist" containerID="e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289" Oct 29 23:54:36.893932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.893903 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289"} err="failed to get container status \"e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289\": rpc error: code = NotFound desc = could not find container \"e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289\": container with ID starting with e180a789eb861568eed797f7d6caae686b6572d3a45751ae75a3313b75729289 not found: ID does not exist" Oct 29 23:54:36.939668 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.936026 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj"] Oct 29 23:54:36.942536 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:36.942503 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-5jmmj"] Oct 29 23:54:37.001910 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:37.001878 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" path="/var/lib/kubelet/pods/f09ce4d3-71b2-458f-ad73-542a86c10268/volumes" Oct 29 23:54:38.678900 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.678873 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:54:38.753037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.753008 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7cc703-61ff-4a81-b215-a5619f21af8a-kube-api-access-d2j4p\" (UniqueName: \"kubernetes.io/projected/ab7cc703-61ff-4a81-b215-a5619f21af8a-kube-api-access-d2j4p\") pod \"ab7cc703-61ff-4a81-b215-a5619f21af8a\" (UID: \"ab7cc703-61ff-4a81-b215-a5619f21af8a\") " Oct 29 23:54:38.753236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.753050 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7cc703-61ff-4a81-b215-a5619f21af8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7cc703-61ff-4a81-b215-a5619f21af8a-plugins-config-volume\") pod \"ab7cc703-61ff-4a81-b215-a5619f21af8a\" (UID: \"ab7cc703-61ff-4a81-b215-a5619f21af8a\") " Oct 29 23:54:38.753445 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.753421 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7cc703-61ff-4a81-b215-a5619f21af8a-plugins-config-volume" pod "ab7cc703-61ff-4a81-b215-a5619f21af8a" (UID: "ab7cc703-61ff-4a81-b215-a5619f21af8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:38.755132 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.755112 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7cc703-61ff-4a81-b215-a5619f21af8a-kube-api-access-d2j4p" pod "ab7cc703-61ff-4a81-b215-a5619f21af8a" (UID: "ab7cc703-61ff-4a81-b215-a5619f21af8a"). InnerVolumeSpecName "kube-api-access-d2j4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:38.854630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.854601 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2j4p\" (UniqueName: \"kubernetes.io/projected/ab7cc703-61ff-4a81-b215-a5619f21af8a-kube-api-access-d2j4p\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:38.854630 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.854627 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab7cc703-61ff-4a81-b215-a5619f21af8a-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:38.891329 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.891302 5736 generic.go:356] "Generic (PLEG): container finished" podID="ab7cc703-61ff-4a81-b215-a5619f21af8a" containerID="a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510" exitCode=0 Oct 29 23:54:38.891456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.891358 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" Oct 29 23:54:38.891456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.891393 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" event={"ID":"ab7cc703-61ff-4a81-b215-a5619f21af8a","Type":"ContainerDied","Data":"a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510"} Oct 29 23:54:38.891615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.891457 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm" event={"ID":"ab7cc703-61ff-4a81-b215-a5619f21af8a","Type":"ContainerDied","Data":"d41a8888bf018a10e9c7812fa60db6c030e873f0118f87748cd639ead3bb450b"} Oct 29 23:54:38.891615 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.891483 5736 scope.go:122] "RemoveContainer" containerID="a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510" Oct 29 23:54:38.902239 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.902219 5736 scope.go:122] "RemoveContainer" containerID="a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510" Oct 29 23:54:38.902503 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:38.902483 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510\": container with ID starting with a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510 not found: ID does not exist" containerID="a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510" Oct 29 23:54:38.902568 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.902515 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510"} err="failed to get container status \"a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510\": rpc error: code = NotFound desc = could not find container \"a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510\": container with ID starting with a99e4781a56d0d76a37edb20f6b4e7e67b87645729cdd9044064aa085bdb1510 not found: ID does not exist" Oct 29 23:54:38.947481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.947458 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm"] Oct 29 23:54:38.951412 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:38.951389 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2126fb-77978f96c9-m55wm"] Oct 29 23:54:39.001888 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:39.001862 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7cc703-61ff-4a81-b215-a5619f21af8a" path="/var/lib/kubelet/pods/ab7cc703-61ff-4a81-b215-a5619f21af8a/volumes" Oct 29 23:54:39.029875 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:39.029809 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55"] Oct 29 23:54:39.033724 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:39.033700 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8"] Oct 29 23:54:39.035717 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:39.035692 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f2056-l9b55"] Oct 29 23:54:39.037115 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:39.037091 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4850bd6-wdkl8"] Oct 29 23:54:41.002080 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:41.002043 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1077baad-3485-4973-951f-50e8ff9d5c0e" path="/var/lib/kubelet/pods/1077baad-3485-4973-951f-50e8ff9d5c0e/volumes" Oct 29 23:54:41.002690 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:41.002668 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedc11e4-3f71-4390-a308-564524688cdd" path="/var/lib/kubelet/pods/bedc11e4-3f71-4390-a308-564524688cdd/volumes" Oct 29 23:54:42.018845 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:42.018807 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg"] Oct 29 23:54:44.300711 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:44.300674 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt"] Oct 29 23:54:44.312405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:44.312371 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-32c192-wzrdt"] Oct 29 23:54:45.001599 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:45.001553 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c264a13b-b2a8-47eb-ac9f-91fd56328700" path="/var/lib/kubelet/pods/c264a13b-b2a8-47eb-ac9f-91fd56328700/volumes" Oct 29 23:54:47.019692 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:47.019631 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" podUID="a127068b-6902-4f90-b90d-c6cc4dfcd6df" containerName="payload-pre-processing" containerID="cri-o://56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa" gracePeriod=25 Oct 29 23:54:48.038764 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:48.038722 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x"] Oct 29 23:54:48.050387 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:48.050355 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03d5a19-rmh5x"] Oct 29 23:54:49.001993 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:49.001949 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64092c48-8906-4330-ae64-9e3b79faba70" path="/var/lib/kubelet/pods/64092c48-8906-4330-ae64-9e3b79faba70/volumes" Oct 29 23:54:51.709248 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:51.709210 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv"] Oct 29 23:54:51.709786 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:51.709467 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" containerID="cri-o://bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba" gracePeriod=30 Oct 29 23:54:54.276405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:54.276365 5736 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:54:54.276803 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:54.276427 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:54:54.793978 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:54.793939 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" podUID="a127068b-6902-4f90-b90d-c6cc4dfcd6df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:54:56.824307 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:56.824269 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:54:56.953046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:56.953021 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 23:54:57.007939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.007868 5736 generic.go:356] "Generic (PLEG): container finished" podID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerID="bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba" exitCode=0 Oct 29 23:54:57.007939 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.007911 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" Oct 29 23:54:57.008112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.007954 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" event={"ID":"e89a6f30-fbab-4350-b331-aa0f2a19b656","Type":"ContainerDied","Data":"bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba"} Oct 29 23:54:57.008112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.008006 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv" event={"ID":"e89a6f30-fbab-4350-b331-aa0f2a19b656","Type":"ContainerDied","Data":"fd0f9644435c87ca7ad642f034c90196f76276cd838fe18fa31b1a03152c3de5"} Oct 29 23:54:57.008112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.008029 5736 scope.go:122] "RemoveContainer" containerID="bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba" Oct 29 23:54:57.020539 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.020509 5736 scope.go:122] "RemoveContainer" containerID="bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba" Oct 29 23:54:57.020848 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:54:57.020828 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba\": container with ID starting with bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba not found: ID does not exist" containerID="bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba" Oct 29 23:54:57.020932 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.020855 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba"} err="failed to get container status \"bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba\": rpc error: code = NotFound desc = could not find container \"bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba\": container with ID starting with bfd79b889eb3c65e97ab4019debce93889485fb646982a118b5da1fb942cd4ba not found: ID does not exist" Oct 29 23:54:57.041321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041294 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-token\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-token\") pod \"e89a6f30-fbab-4350-b331-aa0f2a19b656\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " Oct 29 23:54:57.041452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041350 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-kube-api-access-6cm46\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-kube-api-access-6cm46\") pod \"e89a6f30-fbab-4350-b331-aa0f2a19b656\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " Oct 29 23:54:57.041452 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041423 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89a6f30-fbab-4350-b331-aa0f2a19b656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89a6f30-fbab-4350-b331-aa0f2a19b656-istiod-ca-cert\") pod \"e89a6f30-fbab-4350-b331-aa0f2a19b656\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " Oct 29 23:54:57.041601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041452 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-socket\") pod \"e89a6f30-fbab-4350-b331-aa0f2a19b656\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " Oct 29 23:54:57.041601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041519 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-certs\") pod \"e89a6f30-fbab-4350-b331-aa0f2a19b656\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " Oct 29 23:54:57.041601 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041561 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-data\") pod \"e89a6f30-fbab-4350-b331-aa0f2a19b656\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " Oct 29 23:54:57.041790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041643 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-credential-socket\") pod \"e89a6f30-fbab-4350-b331-aa0f2a19b656\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " Oct 29 23:54:57.041790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041706 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-envoy\") pod \"e89a6f30-fbab-4350-b331-aa0f2a19b656\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " Oct 29 23:54:57.041790 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041743 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-podinfo\") pod \"e89a6f30-fbab-4350-b331-aa0f2a19b656\" (UID: \"e89a6f30-fbab-4350-b331-aa0f2a19b656\") " Oct 29 23:54:57.041967 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041861 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-certs" pod "e89a6f30-fbab-4350-b331-aa0f2a19b656" (UID: "e89a6f30-fbab-4350-b331-aa0f2a19b656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.042028 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.041962 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89a6f30-fbab-4350-b331-aa0f2a19b656-istiod-ca-cert" pod "e89a6f30-fbab-4350-b331-aa0f2a19b656" (UID: "e89a6f30-fbab-4350-b331-aa0f2a19b656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:54:57.042609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.042160 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-data" pod "e89a6f30-fbab-4350-b331-aa0f2a19b656" (UID: "e89a6f30-fbab-4350-b331-aa0f2a19b656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.042609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.042200 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-credential-socket" pod "e89a6f30-fbab-4350-b331-aa0f2a19b656" (UID: "e89a6f30-fbab-4350-b331-aa0f2a19b656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.042609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.042217 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-socket" pod "e89a6f30-fbab-4350-b331-aa0f2a19b656" (UID: "e89a6f30-fbab-4350-b331-aa0f2a19b656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.042609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.042558 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89a6f30-fbab-4350-b331-aa0f2a19b656-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.042609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.042597 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.042609 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.042612 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.042911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.042625 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.042911 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.042637 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.043735 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.043714 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-kube-api-access-6cm46" pod "e89a6f30-fbab-4350-b331-aa0f2a19b656" (UID: "e89a6f30-fbab-4350-b331-aa0f2a19b656"). InnerVolumeSpecName "kube-api-access-6cm46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:57.043811 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.043729 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-token" pod "e89a6f30-fbab-4350-b331-aa0f2a19b656" (UID: "e89a6f30-fbab-4350-b331-aa0f2a19b656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:54:57.044232 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.044211 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-podinfo" pod "e89a6f30-fbab-4350-b331-aa0f2a19b656" (UID: "e89a6f30-fbab-4350-b331-aa0f2a19b656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:54:57.044503 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.044485 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-envoy" pod "e89a6f30-fbab-4350-b331-aa0f2a19b656" (UID: "e89a6f30-fbab-4350-b331-aa0f2a19b656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:54:57.143943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.143913 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cm46\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-kube-api-access-6cm46\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.143943 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.143942 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.144127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.143958 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.144127 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.143970 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89a6f30-fbab-4350-b331-aa0f2a19b656-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:54:57.354376 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.354337 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv"] Oct 29 23:54:57.358521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.358495 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-ccwhv"] Oct 29 23:54:57.955055 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.955018 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf"] Oct 29 23:54:57.961236 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:57.961205 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg"] Oct 29 23:54:59.002183 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:54:59.002146 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" path="/var/lib/kubelet/pods/e89a6f30-fbab-4350-b331-aa0f2a19b656/volumes" Oct 29 23:55:03.027177 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:03.027133 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh"] Oct 29 23:55:03.038186 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:03.038150 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381427e0-gwrlh"] Oct 29 23:55:04.794216 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:04.794179 5736 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" podUID="a127068b-6902-4f90-b90d-c6cc4dfcd6df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:55:05.001963 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:05.001926 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8075cc6b-442f-4d04-bd08-cd5d2e0029f8" path="/var/lib/kubelet/pods/8075cc6b-442f-4d04-bd08-cd5d2e0029f8/volumes" Oct 29 23:55:07.256872 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:07.256846 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:55:07.337252 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:07.337182 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a127068b-6902-4f90-b90d-c6cc4dfcd6df-kube-api-access-lvmh8\" (UniqueName: \"kubernetes.io/projected/a127068b-6902-4f90-b90d-c6cc4dfcd6df-kube-api-access-lvmh8\") pod \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\" (UID: \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\") " Oct 29 23:55:07.337410 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:07.337278 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a127068b-6902-4f90-b90d-c6cc4dfcd6df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a127068b-6902-4f90-b90d-c6cc4dfcd6df-plugins-config-volume\") pod \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\" (UID: \"a127068b-6902-4f90-b90d-c6cc4dfcd6df\") " Oct 29 23:55:07.337660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:07.337635 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a127068b-6902-4f90-b90d-c6cc4dfcd6df-plugins-config-volume" pod "a127068b-6902-4f90-b90d-c6cc4dfcd6df" (UID: "a127068b-6902-4f90-b90d-c6cc4dfcd6df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:55:07.339284 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:07.339260 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a127068b-6902-4f90-b90d-c6cc4dfcd6df-kube-api-access-lvmh8" pod "a127068b-6902-4f90-b90d-c6cc4dfcd6df" (UID: "a127068b-6902-4f90-b90d-c6cc4dfcd6df"). InnerVolumeSpecName "kube-api-access-lvmh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:07.438750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:07.438722 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmh8\" (UniqueName: \"kubernetes.io/projected/a127068b-6902-4f90-b90d-c6cc4dfcd6df-kube-api-access-lvmh8\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:07.438750 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:07.438751 5736 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a127068b-6902-4f90-b90d-c6cc4dfcd6df-plugins-config-volume\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:08.076123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:08.076085 5736 generic.go:356] "Generic (PLEG): container finished" podID="a127068b-6902-4f90-b90d-c6cc4dfcd6df" containerID="56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa" exitCode=0 Oct 29 23:55:08.076343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:08.076136 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" Oct 29 23:55:08.076343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:08.076170 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" event={"ID":"a127068b-6902-4f90-b90d-c6cc4dfcd6df","Type":"ContainerDied","Data":"56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa"} Oct 29 23:55:08.076343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:08.076230 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg" event={"ID":"a127068b-6902-4f90-b90d-c6cc4dfcd6df","Type":"ContainerDied","Data":"3d6ebb7a677d545a930aa0ea8a847f70a7124d4e583302ac2dbee1fcaf5c8761"} Oct 29 23:55:08.076343 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:08.076261 5736 scope.go:122] "RemoveContainer" containerID="56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa" Oct 29 23:55:08.086085 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:08.086067 5736 scope.go:122] "RemoveContainer" containerID="56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa" Oct 29 23:55:08.086343 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:55:08.086322 5736 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa\": container with ID starting with 56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa not found: ID does not exist" containerID="56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa" Oct 29 23:55:08.086456 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:08.086356 5736 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa"} err="failed to get container status \"56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa\": rpc error: code = NotFound desc = could not find container \"56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa\": container with ID starting with 56ca8ef5e57dfb1fc0962f4616b1ed59106c7ef8e3fda7447f25103b7b071bfa not found: ID does not exist" Oct 29 23:55:08.131702 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:08.131670 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg"] Oct 29 23:55:08.143558 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:08.143529 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8a8b61-5c8bdb85c5-hdgsg"] Oct 29 23:55:09.003934 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:09.003899 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a127068b-6902-4f90-b90d-c6cc4dfcd6df" path="/var/lib/kubelet/pods/a127068b-6902-4f90-b90d-c6cc4dfcd6df/volumes" Oct 29 23:55:12.976885 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:12.976813 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" containerID="cri-o://4a2dc493cf8eb0a7e93d227cb1f64eae1e17ebb3eb8d2301f89bd08eecc16dd3" gracePeriod=45 Oct 29 23:55:12.982527 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:12.982495 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" containerID="cri-o://3497f57649ff489e0478acd7bd1d7b078f67ab1cf7e4313fda9f68a409d33491" gracePeriod=45 Oct 29 23:55:13.112639 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.112598 5736 generic.go:356] "Generic (PLEG): container finished" podID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerID="4a2dc493cf8eb0a7e93d227cb1f64eae1e17ebb3eb8d2301f89bd08eecc16dd3" exitCode=0 Oct 29 23:55:13.112916 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.112889 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" event={"ID":"06f33759-614b-4b2c-923f-1c7d238bbb8d","Type":"ContainerDied","Data":"4a2dc493cf8eb0a7e93d227cb1f64eae1e17ebb3eb8d2301f89bd08eecc16dd3"} Oct 29 23:55:13.115018 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.114976 5736 generic.go:356] "Generic (PLEG): container finished" podID="674b738e-397f-4f31-991b-93fdcabf6451" containerID="3497f57649ff489e0478acd7bd1d7b078f67ab1cf7e4313fda9f68a409d33491" exitCode=0 Oct 29 23:55:13.115174 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.115059 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" event={"ID":"674b738e-397f-4f31-991b-93fdcabf6451","Type":"ContainerDied","Data":"3497f57649ff489e0478acd7bd1d7b078f67ab1cf7e4313fda9f68a409d33491"} Oct 29 23:55:13.230571 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.230506 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 23:55:13.233417 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.233394 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 23:55:13.389856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.389817 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-tmp-dir\") pod \"06f33759-614b-4b2c-923f-1c7d238bbb8d\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " Oct 29 23:55:13.390065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.389866 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-home\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-home\") pod \"06f33759-614b-4b2c-923f-1c7d238bbb8d\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " Oct 29 23:55:13.390065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.389913 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674b738e-397f-4f31-991b-93fdcabf6451-kube-api-access-qvvc2\" (UniqueName: \"kubernetes.io/projected/674b738e-397f-4f31-991b-93fdcabf6451-kube-api-access-qvvc2\") pod \"674b738e-397f-4f31-991b-93fdcabf6451\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " Oct 29 23:55:13.390065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.389971 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06f33759-614b-4b2c-923f-1c7d238bbb8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/06f33759-614b-4b2c-923f-1c7d238bbb8d-tls-certs\") pod \"06f33759-614b-4b2c-923f-1c7d238bbb8d\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " Oct 29 23:55:13.390065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390026 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-model-cache\") pod \"06f33759-614b-4b2c-923f-1c7d238bbb8d\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " Oct 29 23:55:13.390065 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390047 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-model-cache\") pod \"674b738e-397f-4f31-991b-93fdcabf6451\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " Oct 29 23:55:13.390397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390072 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f33759-614b-4b2c-923f-1c7d238bbb8d-kube-api-access-p5fnj\" (UniqueName: \"kubernetes.io/projected/06f33759-614b-4b2c-923f-1c7d238bbb8d-kube-api-access-p5fnj\") pod \"06f33759-614b-4b2c-923f-1c7d238bbb8d\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " Oct 29 23:55:13.390397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390098 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-home\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-home\") pod \"674b738e-397f-4f31-991b-93fdcabf6451\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " Oct 29 23:55:13.390397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390124 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/674b738e-397f-4f31-991b-93fdcabf6451-tls-certs\" (UniqueName: \"kubernetes.io/secret/674b738e-397f-4f31-991b-93fdcabf6451-tls-certs\") pod \"674b738e-397f-4f31-991b-93fdcabf6451\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " Oct 29 23:55:13.390397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390149 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-dshm\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-dshm\") pod \"674b738e-397f-4f31-991b-93fdcabf6451\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " Oct 29 23:55:13.390397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390181 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-home" pod "06f33759-614b-4b2c-923f-1c7d238bbb8d" (UID: "06f33759-614b-4b2c-923f-1c7d238bbb8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.390397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390192 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-tmp-dir\") pod \"674b738e-397f-4f31-991b-93fdcabf6451\" (UID: \"674b738e-397f-4f31-991b-93fdcabf6451\") " Oct 29 23:55:13.390397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390318 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-model-cache" pod "06f33759-614b-4b2c-923f-1c7d238bbb8d" (UID: "06f33759-614b-4b2c-923f-1c7d238bbb8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.390397 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390387 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-tmp-dir" pod "674b738e-397f-4f31-991b-93fdcabf6451" (UID: "674b738e-397f-4f31-991b-93fdcabf6451"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.390936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390539 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-home" pod "674b738e-397f-4f31-991b-93fdcabf6451" (UID: "674b738e-397f-4f31-991b-93fdcabf6451"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.390936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390556 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-tmp-dir" pod "06f33759-614b-4b2c-923f-1c7d238bbb8d" (UID: "06f33759-614b-4b2c-923f-1c7d238bbb8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.390936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390653 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-dshm\") pod \"06f33759-614b-4b2c-923f-1c7d238bbb8d\" (UID: \"06f33759-614b-4b2c-923f-1c7d238bbb8d\") " Oct 29 23:55:13.390936 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.390786 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-model-cache" pod "674b738e-397f-4f31-991b-93fdcabf6451" (UID: "674b738e-397f-4f31-991b-93fdcabf6451"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.391172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.391067 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.391172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.391091 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.391172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.391105 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.391172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.391118 5736 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-model-cache\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.391172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.391131 5736 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-home\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.391172 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.391143 5736 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-tmp-dir\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.392650 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.392621 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674b738e-397f-4f31-991b-93fdcabf6451-kube-api-access-qvvc2" pod "674b738e-397f-4f31-991b-93fdcabf6451" (UID: "674b738e-397f-4f31-991b-93fdcabf6451"). InnerVolumeSpecName "kube-api-access-qvvc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:13.392887 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.392839 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f33759-614b-4b2c-923f-1c7d238bbb8d-tls-certs" pod "06f33759-614b-4b2c-923f-1c7d238bbb8d" (UID: "06f33759-614b-4b2c-923f-1c7d238bbb8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:55:13.393026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.392937 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-dshm" pod "674b738e-397f-4f31-991b-93fdcabf6451" (UID: "674b738e-397f-4f31-991b-93fdcabf6451"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.393026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.392959 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/674b738e-397f-4f31-991b-93fdcabf6451-tls-certs" pod "674b738e-397f-4f31-991b-93fdcabf6451" (UID: "674b738e-397f-4f31-991b-93fdcabf6451"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:55:13.393569 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.393546 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-dshm" pod "06f33759-614b-4b2c-923f-1c7d238bbb8d" (UID: "06f33759-614b-4b2c-923f-1c7d238bbb8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:13.393682 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.393565 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f33759-614b-4b2c-923f-1c7d238bbb8d-kube-api-access-p5fnj" pod "06f33759-614b-4b2c-923f-1c7d238bbb8d" (UID: "06f33759-614b-4b2c-923f-1c7d238bbb8d"). InnerVolumeSpecName "kube-api-access-p5fnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:13.491672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.491573 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvvc2\" (UniqueName: \"kubernetes.io/projected/674b738e-397f-4f31-991b-93fdcabf6451-kube-api-access-qvvc2\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.491672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.491625 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f33759-614b-4b2c-923f-1c7d238bbb8d-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.491672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.491636 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5fnj\" (UniqueName: \"kubernetes.io/projected/06f33759-614b-4b2c-923f-1c7d238bbb8d-kube-api-access-p5fnj\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.491672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.491644 5736 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/674b738e-397f-4f31-991b-93fdcabf6451-tls-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.491672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.491653 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/674b738e-397f-4f31-991b-93fdcabf6451-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:13.491672 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:13.491660 5736 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f33759-614b-4b2c-923f-1c7d238bbb8d-dshm\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:14.122254 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.122218 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" Oct 29 23:55:14.122767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.122247 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg" event={"ID":"674b738e-397f-4f31-991b-93fdcabf6451","Type":"ContainerDied","Data":"d88051e83818535542e95a5cc6d9d39105c64a280ac1854782e4af00779db85b"} Oct 29 23:55:14.122767 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.122310 5736 scope.go:122] "RemoveContainer" containerID="3497f57649ff489e0478acd7bd1d7b078f67ab1cf7e4313fda9f68a409d33491" Oct 29 23:55:14.123891 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.123862 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" Oct 29 23:55:14.124026 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.123909 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf" event={"ID":"06f33759-614b-4b2c-923f-1c7d238bbb8d","Type":"ContainerDied","Data":"9a6da5d1a20ac3261727827bbb54678ada69248c391a5be0f0ddde81da0127b1"} Oct 29 23:55:14.135958 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.135937 5736 scope.go:122] "RemoveContainer" containerID="4a2dc493cf8eb0a7e93d227cb1f64eae1e17ebb3eb8d2301f89bd08eecc16dd3" Oct 29 23:55:14.186458 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.186432 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg"] Oct 29 23:55:14.198326 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.198302 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/simulator-w1-99dfb0-kserve-677984f898-dwdwg"] Oct 29 23:55:14.231128 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.231106 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf"] Oct 29 23:55:14.235481 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:14.235459 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/premium-w1-99dfb0-kserve-795bb9db5f-bp5xf"] Oct 29 23:55:15.001838 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:15.001800 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" path="/var/lib/kubelet/pods/06f33759-614b-4b2c-923f-1c7d238bbb8d/volumes" Oct 29 23:55:15.002268 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:15.002253 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674b738e-397f-4f31-991b-93fdcabf6451" path="/var/lib/kubelet/pods/674b738e-397f-4f31-991b-93fdcabf6451/volumes" Oct 29 23:55:26.823204 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:55:26.823165 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:55:27.106498 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:27.106406 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z"] Oct 29 23:55:27.106965 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:27.106934 5736 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" containerID="cri-o://0935a22fd197706b18f506318137970cff11bf752c2f16ad35cfbe3b68e2730b" gracePeriod=30 Oct 29 23:55:31.946489 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:31.946446 5736 scope.go:122] "RemoveContainer" containerID="c84c7418664f41733bcb4921dd2e2370510cc6be92b8134215fa8fc9bc5d691d" Oct 29 23:55:31.956042 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:31.956020 5736 scope.go:122] "RemoveContainer" containerID="d489217f5a7a4aa63c5f615f0c2d93c2ec89cdc977ce31029b95631379f8cfb6" Oct 29 23:55:31.969064 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:31.969040 5736 scope.go:122] "RemoveContainer" containerID="dbb1bd8160825681eba73d5941685ef34ce07ff19cac9a1b9accf23aecc57d55" Oct 29 23:55:31.977806 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:31.977771 5736 scope.go:122] "RemoveContainer" containerID="542fb67b86d69eda7ad0840258d82bcc5337d75e56c2ea2ed3ad39aa4e0f3d9c" Oct 29 23:55:31.985973 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:31.985953 5736 scope.go:122] "RemoveContainer" containerID="831d96c0b5433fb6a6292a7490960f90c53389a114ba6d390adbfa04bac06f7f" Oct 29 23:55:32.245285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.245252 5736 generic.go:356] "Generic (PLEG): container finished" podID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerID="0935a22fd197706b18f506318137970cff11bf752c2f16ad35cfbe3b68e2730b" exitCode=0 Oct 29 23:55:32.245463 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.245305 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" event={"ID":"e0ea4ab6-c8bb-4388-a2a4-44e95490750d","Type":"ContainerDied","Data":"0935a22fd197706b18f506318137970cff11bf752c2f16ad35cfbe3b68e2730b"} Oct 29 23:55:32.359305 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.359279 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 23:55:32.555738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555661 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-data\") pod \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " Oct 29 23:55:32.555738 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555702 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-podinfo\") pod \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " Oct 29 23:55:32.555986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555742 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-credential-socket\") pod \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " Oct 29 23:55:32.555986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555765 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-envoy\") pod \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " Oct 29 23:55:32.555986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555782 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-certs\") pod \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " Oct 29 23:55:32.555986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555805 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-socket\") pod \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " Oct 29 23:55:32.555986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555833 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-kube-api-access-bllkc\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-kube-api-access-bllkc\") pod \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " Oct 29 23:55:32.555986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555857 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istiod-ca-cert\") pod \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " Oct 29 23:55:32.555986 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555909 5736 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-token\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-token\") pod \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\" (UID: \"e0ea4ab6-c8bb-4388-a2a4-44e95490750d\") " Oct 29 23:55:32.556383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.555999 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-data" pod "e0ea4ab6-c8bb-4388-a2a4-44e95490750d" (UID: "e0ea4ab6-c8bb-4388-a2a4-44e95490750d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:32.556383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.556047 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-credential-socket" pod "e0ea4ab6-c8bb-4388-a2a4-44e95490750d" (UID: "e0ea4ab6-c8bb-4388-a2a4-44e95490750d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:32.556383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.556205 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-socket" pod "e0ea4ab6-c8bb-4388-a2a4-44e95490750d" (UID: "e0ea4ab6-c8bb-4388-a2a4-44e95490750d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:32.556383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.556221 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-certs" pod "e0ea4ab6-c8bb-4388-a2a4-44e95490750d" (UID: "e0ea4ab6-c8bb-4388-a2a4-44e95490750d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:32.556383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.556362 5736 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-data\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:32.556383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.556368 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istiod-ca-cert" pod "e0ea4ab6-c8bb-4388-a2a4-44e95490750d" (UID: "e0ea4ab6-c8bb-4388-a2a4-44e95490750d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:55:32.556383 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.556385 5736 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-credential-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:32.556853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.556400 5736 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-certs\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:32.556853 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.556413 5736 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-workload-socket\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:32.558113 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.558080 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-envoy" pod "e0ea4ab6-c8bb-4388-a2a4-44e95490750d" (UID: "e0ea4ab6-c8bb-4388-a2a4-44e95490750d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:55:32.558246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.558126 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-kube-api-access-bllkc" pod "e0ea4ab6-c8bb-4388-a2a4-44e95490750d" (UID: "e0ea4ab6-c8bb-4388-a2a4-44e95490750d"). InnerVolumeSpecName "kube-api-access-bllkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:32.558246 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.558167 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-podinfo" pod "e0ea4ab6-c8bb-4388-a2a4-44e95490750d" (UID: "e0ea4ab6-c8bb-4388-a2a4-44e95490750d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:55:32.558378 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.558361 5736 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-token" pod "e0ea4ab6-c8bb-4388-a2a4-44e95490750d" (UID: "e0ea4ab6-c8bb-4388-a2a4-44e95490750d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:55:32.657856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.657826 5736 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-podinfo\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:32.657856 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.657852 5736 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-envoy\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:32.658029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.657861 5736 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bllkc\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-kube-api-access-bllkc\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:32.658029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.657872 5736 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istiod-ca-cert\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:32.658029 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:32.657881 5736 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ea4ab6-c8bb-4388-a2a4-44e95490750d-istio-token\") on node \"ip-10-0-143-172.ec2.internal\" DevicePath \"\"" Oct 29 23:55:33.251979 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:33.251951 5736 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" Oct 29 23:55:33.252493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:33.251981 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z" event={"ID":"e0ea4ab6-c8bb-4388-a2a4-44e95490750d","Type":"ContainerDied","Data":"4c8e741b582ef55837bc7af9f17db0dc331d07f25b7c6832e5d1df773b94d672"} Oct 29 23:55:33.252493 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:33.252041 5736 scope.go:122] "RemoveContainer" containerID="0935a22fd197706b18f506318137970cff11bf752c2f16ad35cfbe3b68e2730b" Oct 29 23:55:33.315495 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:33.315463 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z"] Oct 29 23:55:33.327289 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:33.327256 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-c9m4z"] Oct 29 23:55:35.002896 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:35.002801 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" path="/var/lib/kubelet/pods/e0ea4ab6-c8bb-4388-a2a4-44e95490750d/volumes" Oct 29 23:55:36.037398 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:36.037351 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn"] Oct 29 23:55:36.049960 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:36.049922 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269a18c6-kwwkn"] Oct 29 23:55:37.001105 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:37.001072 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351f64d9-561d-49be-a84b-14b91010bcec" path="/var/lib/kubelet/pods/351f64d9-561d-49be-a84b-14b91010bcec/volumes" Oct 29 23:55:41.753237 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:41.753200 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l9tmh_5e30a728-ba3a-4405-8a8f-5b1ec88db691/manager/0.log" Oct 29 23:55:56.823208 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:55:56.823163 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:55:58.030654 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:58.030614 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f"] Oct 29 23:55:58.042475 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:58.042438 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82dade27-c2t5f"] Oct 29 23:55:59.002116 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:55:59.002070 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145e43bb-0faa-4488-8aff-4c3ae43208c1" path="/var/lib/kubelet/pods/145e43bb-0faa-4488-8aff-4c3ae43208c1/volumes" Oct 29 23:56:05.026153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:05.026113 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td"] Oct 29 23:56:05.037976 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:05.037946 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ecfa1a-cs7td"] Oct 29 23:56:05.209592 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:05.209553 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l9tmh_5e30a728-ba3a-4405-8a8f-5b1ec88db691/manager/0.log" Oct 29 23:56:07.002267 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:07.002232 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df8051e-1cd4-45dc-b68e-d4d7875eade9" path="/var/lib/kubelet/pods/7df8051e-1cd4-45dc-b68e-d4d7875eade9/volumes" Oct 29 23:56:07.531405 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:07.531369 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bcf7db76f-srkgv_a1cdd049-d5d5-4a3a-87cb-360491c56536/maas-api/0.log" Oct 29 23:56:07.938490 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:07.938412 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2816861c-wdg2v_8855fdf9-1a85-4364-a2d6-10a9278b9bc4/revoke-keys/0.log" Oct 29 23:56:08.472784 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:08.472756 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-60d903d6-rs5dr_2c2959a1-c03a-42aa-999e-135769ce1efc/revoke-keys/0.log" Oct 29 23:56:08.639061 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:08.639030 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-89c87936-kn9nf_bef6171a-064f-4f9a-a65d-6fed44a00d36/revoke-keys/0.log" Oct 29 23:56:08.997774 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:08.997739 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e17667bf-kpwqd_4317d6bc-77d6-4fae-b428-604f09fc2e58/revoke-keys/0.log" Oct 29 23:56:09.188863 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:09.188833 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ea0a4246-f8p8d_e8fb7bea-c53a-4cef-9400-c87c06b54554/revoke-keys/0.log" Oct 29 23:56:09.378768 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:09.378686 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-x4z7h_c9f3a369-0f84-4d09-bd71-180facff85a0/postgres/0.log" Oct 29 23:56:13.792117 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:13.792080 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-p6cwx_ba2b1209-5252-4af5-9ba1-e47404585d06/discovery/0.log" Oct 29 23:56:13.984258 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:13.984221 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67f887f896-8g7tq_323481e4-8d92-49db-aef1-5ac5b469271e/kube-auth-proxy/0.log" Oct 29 23:56:24.338156 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:24.338118 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-2dlnz_98a25bcb-0734-4d38-a889-214cfcf54e8b/istio-proxy/0.log" Oct 29 23:56:26.822371 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:56:26.822335 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:56:27.026737 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:27.026700 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf"] Oct 29 23:56:27.038531 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:27.038502 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c87936-kn9nf"] Oct 29 23:56:27.673349 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:27.673314 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-blz4g_ff018793-fe72-4680-af0f-6a7b1d4b94b1/main/0.log" Oct 29 23:56:29.002336 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:29.002299 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef6171a-064f-4f9a-a65d-6fed44a00d36" path="/var/lib/kubelet/pods/bef6171a-064f-4f9a-a65d-6fed44a00d36/volumes" Oct 29 23:56:32.058092 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:32.058061 5736 scope.go:122] "RemoveContainer" containerID="11979fb2ea105e308668aef91adb14520f54a21d894726eb391a99164378c9eb" Oct 29 23:56:32.067816 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:32.067789 5736 scope.go:122] "RemoveContainer" containerID="98bc183e98780b5426e732780c4782655e7511603da563d5061146c97b18ee88" Oct 29 23:56:32.075849 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:32.075815 5736 scope.go:122] "RemoveContainer" containerID="5f659e85217e5f6793be2aab436f5f1481bb796a63fcbaa6e443f47e970130bc" Oct 29 23:56:32.083502 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:32.083481 5736 scope.go:122] "RemoveContainer" containerID="9e15d4990d9ee713058fb6d709924a946bc0ec590709532b8181a321a20ad2a2" Oct 29 23:56:32.091075 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:56:32.091045 5736 scope.go:122] "RemoveContainer" containerID="ea2d9879c8b7370e7f83d5bb803d788ce8a4bc87ae33500240553b99e48338e6" Oct 29 23:56:56.822850 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:56:56.822806 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:57:04.211094 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:04.210996 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-twb2h_fb913afd-66ca-46e3-b9d1-66cff498245d/global-pull-secret-syncer/0.log" Oct 29 23:57:04.919518 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:04.919482 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lgqrb_4fd11cc5-35ab-4347-8914-e953b71bc322/konnectivity-agent/0.log" Oct 29 23:57:05.577465 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:05.577434 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-172.ec2.internal_f3d32c15ddf0c3d5f6df2743c0d67b79/haproxy/0.log" Oct 29 23:57:14.026434 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:14.026398 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d"] Oct 29 23:57:14.038358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:14.038324 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a4246-f8p8d"] Oct 29 23:57:14.039266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:14.039253 5736 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr"] Oct 29 23:57:14.040079 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:14.040055 5736 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d903d6-rs5dr"] Oct 29 23:57:15.001746 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:15.001708 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2959a1-c03a-42aa-999e-135769ce1efc" path="/var/lib/kubelet/pods/2c2959a1-c03a-42aa-999e-135769ce1efc/volumes" Oct 29 23:57:15.002129 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:15.002115 5736 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8fb7bea-c53a-4cef-9400-c87c06b54554" path="/var/lib/kubelet/pods/e8fb7bea-c53a-4cef-9400-c87c06b54554/volumes" Oct 29 23:57:15.374043 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:15.373926 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9eb7d909-e423-4efb-89ba-97cddc92628e/alertmanager/0.log" Oct 29 23:57:15.490658 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:15.490625 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9eb7d909-e423-4efb-89ba-97cddc92628e/config-reloader/0.log" Oct 29 23:57:15.604208 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:15.604170 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9eb7d909-e423-4efb-89ba-97cddc92628e/kube-rbac-proxy-web/0.log" Oct 29 23:57:15.725492 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:15.725461 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9eb7d909-e423-4efb-89ba-97cddc92628e/kube-rbac-proxy/0.log" Oct 29 23:57:15.838861 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:15.838833 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9eb7d909-e423-4efb-89ba-97cddc92628e/kube-rbac-proxy-metric/0.log" Oct 29 23:57:15.955392 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:15.955361 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9eb7d909-e423-4efb-89ba-97cddc92628e/prom-label-proxy/0.log" Oct 29 23:57:16.004854 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.004723 5736 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l"] Oct 29 23:57:16.005209 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005180 5736 cpu_manager.go:414] "containerMap: removing container" podUID="a127068b-6902-4f90-b90d-c6cc4dfcd6df" containerName="payload-pre-processing" Oct 29 23:57:16.005316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005211 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a127068b-6902-4f90-b90d-c6cc4dfcd6df" containerName="payload-pre-processing" Oct 29 23:57:16.005316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005230 5736 cpu_manager.go:414] "containerMap: removing container" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" Oct 29 23:57:16.005316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005238 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" Oct 29 23:57:16.005316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005257 5736 cpu_manager.go:414] "containerMap: removing container" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" Oct 29 23:57:16.005316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005266 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" Oct 29 23:57:16.005316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005280 5736 cpu_manager.go:414] "containerMap: removing container" podUID="8855fdf9-1a85-4364-a2d6-10a9278b9bc4" containerName="revoke-keys" Oct 29 23:57:16.005316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005287 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8855fdf9-1a85-4364-a2d6-10a9278b9bc4" containerName="revoke-keys" Oct 29 23:57:16.005316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005303 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" Oct 29 23:57:16.005316 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005312 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005322 5736 cpu_manager.go:414] "containerMap: removing container" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerName="istio-proxy" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005329 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerName="istio-proxy" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005341 5736 cpu_manager.go:414] "containerMap: removing container" podUID="ab7cc703-61ff-4a81-b215-a5619f21af8a" containerName="payload-pre-processing" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005350 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7cc703-61ff-4a81-b215-a5619f21af8a" containerName="payload-pre-processing" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005376 5736 cpu_manager.go:414] "containerMap: removing container" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005382 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005403 5736 cpu_manager.go:414] "containerMap: removing container" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005410 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005423 5736 cpu_manager.go:414] "containerMap: removing container" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005432 5736 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005518 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f33759-614b-4b2c-923f-1c7d238bbb8d" containerName="main" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005536 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="674b738e-397f-4f31-991b-93fdcabf6451" containerName="main" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005547 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="8855fdf9-1a85-4364-a2d6-10a9278b9bc4" containerName="revoke-keys" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005557 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89a6f30-fbab-4350-b331-aa0f2a19b656" containerName="istio-proxy" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005568 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09ce4d3-71b2-458f-ad73-542a86c10268" containerName="istio-proxy" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005595 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7cc703-61ff-4a81-b215-a5619f21af8a" containerName="payload-pre-processing" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005609 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="902df38e-ddc0-457e-8877-2702115f3de8" containerName="main" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005622 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ea4ab6-c8bb-4388-a2a4-44e95490750d" containerName="istio-proxy" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005631 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea31536-fd5e-489b-8092-8d9a8cb816cd" containerName="main" Oct 29 23:57:16.005954 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.005642 5736 memory_manager.go:365] "RemoveStaleState removing state" podUID="a127068b-6902-4f90-b90d-c6cc4dfcd6df" containerName="payload-pre-processing" Oct 29 23:57:16.009213 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.009182 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.023819 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.023793 5736 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t99pv\"/\"default-dockercfg-8lgdb\"" Oct 29 23:57:16.029624 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.029603 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l"] Oct 29 23:57:16.076998 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.076944 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9eb7d909-e423-4efb-89ba-97cddc92628e/init-config-reloader/0.log" Oct 29 23:57:16.105040 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.105015 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t99pv\"/\"kube-root-ca.crt\"" Oct 29 23:57:16.105138 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.105114 5736 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t99pv\"/\"openshift-service-ca.crt\"" Oct 29 23:57:16.137905 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.137878 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6tst\" (UniqueName: \"kubernetes.io/projected/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-kube-api-access-f6tst\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.138008 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.137992 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-podres\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.138067 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.138036 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-sys\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.138285 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.138266 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-lib-modules\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.138328 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.138299 5736 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-proc\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.239521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.239492 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-lib-modules\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.239521 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.239522 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-proc\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.239715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.239549 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6tst\" (UniqueName: \"kubernetes.io/projected/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-kube-api-access-f6tst\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.239715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.239632 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-podres\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.239715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.239646 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-lib-modules\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.239715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.239647 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-proc\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.239715 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.239677 5736 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-sys\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.239935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.239742 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-podres\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.239935 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.239770 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-sys\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.280358 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.280300 5736 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6tst\" (UniqueName: \"kubernetes.io/projected/7c28dca9-e2bf-4116-afae-d9e2ffef73ad-kube-api-access-f6tst\") pod \"perf-node-gather-daemonset-mws2l\" (UID: \"7c28dca9-e2bf-4116-afae-d9e2ffef73ad\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.321914 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.321889 5736 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.404851 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.404821 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7lhcf_875f6c48-91a9-4f0b-a71d-e41d02e589d5/kube-state-metrics/0.log" Oct 29 23:57:16.510200 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.510174 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7lhcf_875f6c48-91a9-4f0b-a71d-e41d02e589d5/kube-rbac-proxy-main/0.log" Oct 29 23:57:16.617947 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.617876 5736 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l"] Oct 29 23:57:16.622850 ip-10-0-143-172 kubenswrapper[5736]: W1029 23:57:16.622815 5736 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7c28dca9_e2bf_4116_afae_d9e2ffef73ad.slice/crio-47347fc071f41b61a50e87522187f7a00c1081baeaf916897616fd3c70b7632a WatchSource:0}: Error finding container 47347fc071f41b61a50e87522187f7a00c1081baeaf916897616fd3c70b7632a: Status 404 returned error can't find the container with id 47347fc071f41b61a50e87522187f7a00c1081baeaf916897616fd3c70b7632a Oct 29 23:57:16.623590 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.623556 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7lhcf_875f6c48-91a9-4f0b-a71d-e41d02e589d5/kube-rbac-proxy-self/0.log" Oct 29 23:57:16.742321 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.742292 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-75f497d78f-wzp7l_1e4f37ae-5582-442e-bf04-a2cc29932ec6/metrics-server/0.log" Oct 29 23:57:16.824520 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.824483 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" event={"ID":"7c28dca9-e2bf-4116-afae-d9e2ffef73ad","Type":"ContainerStarted","Data":"cb24a4c9a512af7623f3ce3f1c7479f20c458f638af19333c090418fe592acf3"} Oct 29 23:57:16.824695 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.824607 5736 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" event={"ID":"7c28dca9-e2bf-4116-afae-d9e2ffef73ad","Type":"ContainerStarted","Data":"47347fc071f41b61a50e87522187f7a00c1081baeaf916897616fd3c70b7632a"} Oct 29 23:57:16.824762 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.824735 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:16.865686 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.865638 5736 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" podStartSLOduration=1.8656222260000002 podStartE2EDuration="1.865622226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:57:16.864566341 +0000 UTC m=+4433.980485948" watchObservedRunningTime="2026-10-29 23:57:16.865622226 +0000 UTC m=+4433.981541827" Oct 29 23:57:16.869338 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:16.869274 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-4l2vb_bb967bfd-fb93-4051-bda5-9dbdf3589a69/monitoring-plugin/0.log" Oct 29 23:57:17.741670 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:17.741635 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rc9cg_b1d25bfd-dba6-4268-8123-533f4b106c0a/node-exporter/0.log" Oct 29 23:57:17.855677 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:17.855639 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rc9cg_b1d25bfd-dba6-4268-8123-533f4b106c0a/kube-rbac-proxy/0.log" Oct 29 23:57:17.973170 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:17.973128 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rc9cg_b1d25bfd-dba6-4268-8123-533f4b106c0a/init-textfile/0.log" Oct 29 23:57:18.099610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:18.099467 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zx4x7_8cdf7ea9-aed7-4071-ada6-2f0518de3a87/kube-rbac-proxy-main/0.log" Oct 29 23:57:18.216153 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:18.216112 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zx4x7_8cdf7ea9-aed7-4071-ada6-2f0518de3a87/kube-rbac-proxy-self/0.log" Oct 29 23:57:18.333949 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:18.333894 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-zx4x7_8cdf7ea9-aed7-4071-ada6-2f0518de3a87/openshift-state-metrics/0.log" Oct 29 23:57:18.347442 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:18.347413 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l9tmh_5e30a728-ba3a-4405-8a8f-5b1ec88db691/manager/0.log" Oct 29 23:57:18.542162 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:18.542120 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l9tmh_5e30a728-ba3a-4405-8a8f-5b1ec88db691/manager/0.log" Oct 29 23:57:19.705413 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:19.705379 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76865c6696-5hm7s_9009d30d-e890-4365-83af-2244164260f3/telemeter-client/0.log" Oct 29 23:57:19.824034 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:19.824006 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76865c6696-5hm7s_9009d30d-e890-4365-83af-2244164260f3/reload/0.log" Oct 29 23:57:19.940123 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:19.940089 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76865c6696-5hm7s_9009d30d-e890-4365-83af-2244164260f3/kube-rbac-proxy/0.log" Oct 29 23:57:20.064255 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:20.064223 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c4948c74-qwn7r_c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/thanos-query/0.log" Oct 29 23:57:20.178629 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:20.178575 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c4948c74-qwn7r_c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/kube-rbac-proxy-web/0.log" Oct 29 23:57:20.292122 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:20.292091 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c4948c74-qwn7r_c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/kube-rbac-proxy/0.log" Oct 29 23:57:20.408319 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:20.408243 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c4948c74-qwn7r_c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/prom-label-proxy/0.log" Oct 29 23:57:20.522057 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:20.522016 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c4948c74-qwn7r_c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/kube-rbac-proxy-rules/0.log" Oct 29 23:57:20.637416 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:20.637384 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86c4948c74-qwn7r_c9dc5151-8f55-47c4-a3a5-4e063c1e1ef5/kube-rbac-proxy-metrics/0.log" Oct 29 23:57:22.843477 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:22.843404 5736 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-mws2l" Oct 29 23:57:23.511060 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:23.511015 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l9tmh_5e30a728-ba3a-4405-8a8f-5b1ec88db691/manager/0.log" Oct 29 23:57:23.567382 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:23.567348 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l9tmh_5e30a728-ba3a-4405-8a8f-5b1ec88db691/manager/0.log" Oct 29 23:57:26.822613 ip-10-0-143-172 kubenswrapper[5736]: E1029 23:57:26.822558 5736 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zcdnt" containerName="extractor" Oct 29 23:57:29.194409 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:29.194371 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l9tmh_5e30a728-ba3a-4405-8a8f-5b1ec88db691/manager/0.log" Oct 29 23:57:29.300834 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:29.300778 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l9tmh_5e30a728-ba3a-4405-8a8f-5b1ec88db691/manager/0.log" Oct 29 23:57:29.440610 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:29.440551 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q7trl_d31f1eee-73b6-4b6d-bde0-1006646b0173/dns/0.log" Oct 29 23:57:29.556037 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:29.556008 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q7trl_d31f1eee-73b6-4b6d-bde0-1006646b0173/kube-rbac-proxy/0.log" Oct 29 23:57:29.799843 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:29.799815 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5rgg6_8b726c8b-651c-4d5e-afa5-26a9f159d609/dns-node-resolver/0.log" Oct 29 23:57:31.572290 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:31.572253 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ncjz6_878dc633-6f09-450f-ab79-8788f14e3055/node-ca/0.log" Oct 29 23:57:32.135455 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:32.135415 5736 scope.go:122] "RemoveContainer" containerID="712963b91d66ee15e473ee21c8fa43b1accc7dded6460da68c0a06d330e33157" Oct 29 23:57:32.152302 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:32.152278 5736 scope.go:122] "RemoveContainer" containerID="7e4318ec02f6bf8fc94e3b31899eae41e97281ab340ebc032bd0da15c86a5295" Oct 29 23:57:32.163844 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:32.163810 5736 scope.go:122] "RemoveContainer" containerID="9a137bb5e483abefa43d9823719ed903b3d6d206946a53e9f99631d9d396957e" Oct 29 23:57:32.173894 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:32.173873 5736 scope.go:122] "RemoveContainer" containerID="a260acd4b1fe35f6562ed969939d561b0bdd158508f859f80f5c9a84362fb5d2" Oct 29 23:57:32.185660 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:32.185638 5736 scope.go:122] "RemoveContainer" containerID="f6299d31ede4ad429cbc1fbf51ce5701ea1a5869616e7547eee17a37b6369d78" Oct 29 23:57:32.196157 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:32.196125 5736 scope.go:122] "RemoveContainer" containerID="196bdb878955a2d07730e1020ae66ec89f7e703b9be288f378ca6cb2fc6d58b5" Oct 29 23:57:32.206022 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:32.205997 5736 scope.go:122] "RemoveContainer" containerID="b6942a3fd581494cf9da296491f81ed6d226d6de95cc43b3fac852b2db6de5b2" Oct 29 23:57:33.908090 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:33.908051 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-p6cwx_ba2b1209-5252-4af5-9ba1-e47404585d06/discovery/0.log" Oct 29 23:57:33.947112 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:33.947079 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67f887f896-8g7tq_323481e4-8d92-49db-aef1-5ac5b469271e/kube-auth-proxy/0.log" Oct 29 23:57:34.077448 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:34.077415 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-2dlnz_98a25bcb-0734-4d38-a889-214cfcf54e8b/istio-proxy/0.log" Oct 29 23:57:35.880046 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:35.880006 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-p6cwx_ba2b1209-5252-4af5-9ba1-e47404585d06/discovery/0.log" Oct 29 23:57:35.923204 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:35.923178 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67f887f896-8g7tq_323481e4-8d92-49db-aef1-5ac5b469271e/kube-auth-proxy/0.log" Oct 29 23:57:36.015380 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:36.015342 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-p6cwx_ba2b1209-5252-4af5-9ba1-e47404585d06/discovery/0.log" Oct 29 23:57:36.049155 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:36.049121 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67f887f896-8g7tq_323481e4-8d92-49db-aef1-5ac5b469271e/kube-auth-proxy/0.log" Oct 29 23:57:36.056266 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:36.056231 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-2dlnz_98a25bcb-0734-4d38-a889-214cfcf54e8b/istio-proxy/0.log" Oct 29 23:57:36.183125 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:36.183037 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-2dlnz_98a25bcb-0734-4d38-a889-214cfcf54e8b/istio-proxy/0.log" Oct 29 23:57:36.359951 ip-10-0-143-172 kubenswrapper[5736]: I1029 23:57:36.359916 5736 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-p6cwx_ba2b1209-5252-4af5-9ba1-e47404585d06/discovery/0.log"